AWS lambda 应用程序的 python 依赖项已超过 AWS Lambda 的 250 MB 限制。这些依赖项之一是 rasterio,它依赖于 gdal。我正在尝试构建一个 docker 映像来绕过 250 MB 限制并将我们的代码部署到 AWS Lambda(使用 serverless.com)。
方法一:pip install rasterio
目前我有一个Dockerfile
:
FROM public.ecr.aws/lambda/python:3.10
RUN pip install rasterio # Fails with error (see below)
WARNING:root:Failed to get options via gdal-config: [Errno 2] No such file or directory: 'gdal-config'
ERROR: A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable.
方法 2: yum install gdal-devel
TL; dr:“没有可用的 gdal-devel 软件包。”
方法3:构建gdal
TL; 博士:有很多依赖性。令人紧张的是,这些依赖项也将有需要构建的依赖项。
方法 4: yum install epel-release 然后 gdal-devel
- 需要 fortran:
yum -y install libgfortran
可以工作但已安装 libgfortran.so.4 yum -y install gdal-devel
仍然出错,例如“错误:软件包:openblas-openmp-0.3.3-2.el7.aarch64(epel)需要:libgfortran.so.3(GFORTRAN_1.0)(64位)”- 我不确定问题是使用版本 4 而不是 libgfortran 版本 3,但我无法轻松安装
libgfortran.so.3
。
方法5:使用aws/sam/build-python
容器
aws/sam/build-python
容器使用serverless-python-requirements即使用以下无服务器运行时,python 依赖项安装没有问题(没有 rasterio 安装没有错误,但在 AWS 上运行时 lambda 失败,并出现错误“没有名为‘rasterio._version’的模块” )。 yml 文件:serverless.com
serverless deploy
service: aws-python-docker-demo
frameworkVersion: "3"
plugins:
- serverless-python-requirements
custom:
pythonRequirements:
usePipenv: true
layer: true
provider:
name: aws
runtime: python3.10
deploymentBucket:
blockPublicAccess: true
functions:
hello:
handler: src/main.lambda_handler
layers:
- !Ref PythonRequirementsLambdaLayer
这个
serverless-python-requirements
插件似乎使用 docker 容器public.ecr.aws/sam/build-python3.10
来安装 python 依赖项并将它们压缩为 lambda- (然后会失败,因为 lambda 的依赖项和代码 >= 250 MB 大小限制)
计划:了解如何serverless-python-requirements
:public.ecr.aws/sam/build-python3.10
在容器内安装 python 依赖项zips python 依赖项(> 250 MB)
将该 zip 复制到 AWS lambda 的 docker 映像中。...?
我不确定这是否是一个好方法,我确信有更好的解决方案。欢迎任何建议。
**更新**关于新方法(第6号)并回应@Rob的友善回答。
方法 6:尝试使用旧的 gdal/lambda docker 镜像
使用https://hub.docker.com/r/remotepixel/amazonlinux-gdal/进行中的工作。下一步:让它工作,然后从那里迭代到:
- 更新gdal
- 使用最新的 lambda 容器
- 使用 python 3.10(根据我们的应用程序的要求)
目前计划重新回答/更新此 StackOverflow 问题的答案:https://stackoverflow.com/questions/36772111/how-can-i-install-a-recent-version-of-gdal-on-amazon-linux #评论135429542_44907360
目前错误如下:
{
"errorType": "Runtime.InvalidEntrypoint",
"errorMessage": "RequestId: 2cda4291-3b02-4079-8d59-f1ab111f8dab Error: exec: \"main.lambda_handler\": executable file not found in $PATH"
}
回应 Rob 的潜在答案
当我运行它时,出现以下错误:
cat Dockerfile2
FROM public.ecr.aws/lambda/python:3.10
RUN pip install rasterio
docker --version
Docker version 24.0.6, build ed223bc
macOS 12.7.2
docker build -t testing-run-api-dependencies-2 -f ./Dockerfile2 . --progress=plain --no-cache
#0 building with "desktop-linux" instance using docker driver
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile2
#2 transferring dockerfile: 101B done
#2 DONE 0.0s
#3 [internal] load metadata for public.ecr.aws/lambda/python:3.10
#3 DONE 1.1s
#4 [1/2] FROM public.ecr.aws/lambda/python:3.10@sha256:f95780930513037d252b6b6165720381a1014096c3be9f2eac620776c8f0d167
#4 CACHED
#5 [2/2] RUN pip install rasterio
#5 1.173 Collecting rasterio
#5 1.229 Downloading rasterio-1.3.9.tar.gz (411 kB)
#5 1.309 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 411.7/411.7 kB 5.5 MB/s eta 0:00:00
#5 1.406 Installing build dependencies: started
#5 8.663 Installing build dependencies: finished with status 'done'
#5 8.666 Getting requirements to build wheel: started
#5 8.934 Getting requirements to build wheel: finished with status 'error'
#5 8.939 error: subprocess-exited-with-error
#5 8.939
#5 8.939 × Getting requirements to build wheel did not run successfully.
#5 8.939 │ exit code: 1
#5 8.939 ╰─> [3 lines of output]
#5 8.939 <string>:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
#5 8.939 WARNING:root:Failed to get options via gdal-config: [Errno 2] No such file or directory: 'gdal-config'
#5 8.939 ERROR: A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable.
#5 8.939 [end of output]
#5 8.939
#5 8.939 note: This error originates from a subprocess, and is likely not a problem with pip.
#5 8.942 error: subprocess-exited-with-error
#5 8.942
#5 8.942 × Getting requirements to build wheel did not run successfully.
#5 8.942 │ exit code: 1
#5 8.942 ╰─> See above for output.
#5 8.942
#5 8.942 note: This error originates from a subprocess, and is likely not a problem with pip.
#5 8.947
#5 8.947 [notice] A new release of pip is available: 23.0.1 -> 24.0
#5 8.947 [notice] To update, run: pip install --upgrade pip
#5 ERROR: process "/bin/sh -c pip install rasterio" did not complete successfully: exit code: 1
------
> [2/2] RUN pip install rasterio:
8.942 error: subprocess-exited-with-error
8.942
8.942 × Getting requirements to build wheel did not run successfully.
8.942 │ exit code: 1
8.942 ╰─> See above for output.
8.942
8.942 note: This error originates from a subprocess, and is likely not a problem with pip.
8.947
8.947 [notice] A new release of pip is available: 23.0.1 -> 24.0
8.947 [notice] To update, run: pip install --upgrade pip
------
Dockerfile2:2
--------------------
1 | FROM public.ecr.aws/lambda/python:3.10
2 | >>> RUN pip install rasterio
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install rasterio" did not complete successfully: exit code: 1