Amazon EventBridge is a serverless event bus service that makes it easy to connect your applications with data from a variety of sources. You can create rules that self-trigger on an automated schedule in EventBridge using cron or rate expressions. These rules can in-turn trigger Lambda functions, even with custom input if required. All scheduledContinue reading “Run Lambda Functions on a Schedule using Amazon EventBridge Event Rules”

If you’ve ever tried to use an SQS queue in another region as the destination for S3 events, you must have seen this error: The notification destination service region is not valid for the bucket location constraint. What that cryptic error message essentially means is that the S3 bucket & the event destination must beContinue reading “Deliver S3 Events Across Regions by Routing them Through an SNS Topic”

In normal usage, terraform init downloads & installs the plugins for any providers used in the configuration automatically in the .terraform directory. It’s sometimes desirable to disable this behavior, either because you wanna do away with the re-download every time on your dev system, or because you’re running Terraform in a CI/CD pipeline where it’sContinue reading “Use Pre-Installed Terraform Plugins Instead of Downloading them with “terraform init””

If you’re looking for a way to deploy copies of an infrastructure, or parts of an infrastructure to several AWS accounts simultaneously, there’s an easy way to do this. It’s done by using multiple “provider configurations”. As you might be aware, the Terraform provider for AWS must be configured with a way to authenticate itselfContinue reading “Deploy to Multiple AWS Accounts with Terraform”

This post explores ways to structure your Terraform configuration when it’s to be used to deploy infrastructure across multiple cloud accounts, for multiple customers of yours & for multiple environments for each app involved — development, staging, production. One prime example where this might be very useful to you is if you build a multi-tenantContinue reading “Terraform State Management in Multi-Customer Multi-Account Multi-Environment Scenarios”

The upload() method in the AWS JavaScript SDK does a good job of uploading objects to S3 even if they’re large enough to warrant a multipart upload. It’s also possible to pipe a data stream to it in order to upload very large objects. To do this, simply wrap the upload() function with the Node.jsContinue reading “Programmatically Stream (Upload) Large Files to Amazon S3”

According to AWS Glue documentation: Only pure Python libraries can be used. Libraries that rely on C extensions, such as the pandas Python Data Analysis Library, are not yet supported. — Providing Your Own Custom Scripts But if you’re using Python shell jobs in Glue, there is a way to use Python packages like PandasContinue reading “Use Python Packages like NumPy & Pandas with AWS Glue”

If you’re running Spark on EMR & need to submit jobs remotely, you’re in the right place! You can have Airflow running on an EC2 instance & use it to submit jobs to EMR, provided they can reach each other. There are several ways you can trigger a spark-submit to a remote Spark server, EMRContinue reading “Submit Apache Spark Jobs to an Amazon EMR Cluster from Apache Airflow”

RDS Aurora MySQL in AWS provides an in-built feature to create a cross-region read replica of a database. This is easily accessible from the console as shown below: This article describes at a high-level the basic logistics of how replication really happens. First things first: How MySQL Binlog Replication Works The MySQL replication feature allowsContinue reading “How Amazon RDS Aurora MySQL Cross-Region Replication Really Works Under the Hood”

Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. Built by experienced developers, it takes care of much of the hassle of Web development, so you can focus on writing your app without needing to reinvent the wheel. It’s free and open source. Celery is a task queue implementationContinue reading “Run Celery Tasks in the Background with a Django App in AWS Elastic Beanstalk”

If you’re new to AWS (or not), you might run into issues trying to SSH into EC2 instances. This article summarizes things you can try to fix the issue. If you get “Permission denied (publickey)” error or simply aren’t able to SSH to an instance, try the following: Ensure the file permissions on the privateContinue reading “Troubleshooting Permission Denied Errors When Trying to SSH Into an AWS EC2 Instance”

It’s common in many webapps to let the user upload files to your system. These could be as simple as profile pictures or something else. In AWS, the ideal place to keep these files is S3 but if you’re uploading the files to your server & then to S3, you don’t have to. You canContinue reading “Uploading Files to Amazon S3 Directly from a Browser using JavaScript”

Do you have a need to upload a file to an API endpoint in API Gateway? Whether you need to HTTP POST a binary or a text file to an API, a.k.a., multipart form-data, this article will explain how to accomplish this. The way to do this is to define multipart/form-data as a binary mediaContinue reading “POSTing Binary or Multipart Form-Data to an API in Amazon API Gateway”

Are you using .htaccess to configure your Apache server? Or are you trying to use it but nothing you put in there seems to work? When should you use .htaccess instead of editing the server config directly? This article explains the role of .htaccess in Apache server & answers these questions. First thing first —Continue reading “Troubleshooting .htaccess Issues with Apache Server”

There are many ways to redirect incoming HTTP requests to HTTPS in Beanstalk. This article explores a few, with pros & cons of each. But first, the prerequisites: Prerequisites Before we get to Beanstalk itself, there are a few things to take care of, to make HTTPS work. The rest of this article assumes thatContinue reading “Redirect HTTP to HTTPS in AWS Elastic Beanstalk”

There are several ways to make objects in an S3 bucket public. The first is to use the following bucket policy: This (& other) policies can be generated using AWS’s official policy generator at awspolicygen.s3.amazonaws.com/policygen.html: You can also use the (legacy) Access Control Lists to allow public read: If you’re using the S3 CLI toContinue reading “How to Make All Objects in an Amazon S3 Bucket Public by Default?”

Just like you speed-up delivery of your static assets from S3 buckets using CloudFront CDN, you can do the same for your APIs as well. In most cases, it’s enough to make your API “edge-optimized”: If you do this, API Gateway creates & manages a CloudFront distribution for you behind the scenes, but if youContinue reading “Enhance API Client Experience by Deploying a CloudFront Distribution to Serve APIs from Amazon API Gateway”

Follow these steps to set up an SFTP server on an EC2 instance. Start by installing the SFTP server: Allow inbound traffic in your instance’s security group: Edit /etc/vsftpd/vsftpd.conf: Change anonymous_enable=YES to anonymous_enable=NO. Add the following to the end of the file: Restart the SFTP server: or /etc/vsftpd/user_list lists users who are NOT allowed SFTPContinue reading “Run an SFTP Server on an AWS EC2 Instance”

There are several reasons for needing to change (or add) the SSH key pair for your existing EC2 instances. Maybe you lost the private key or you just want to add more keys to the instance so users with those keys can login to it. Let’s see how this can be done. If you lostContinue reading “How to Change the SSH Key Pair for an AWS EC2 Instance?”

Your Beanstalk app runs on one or more EC2 instances managed by AWS behind the scenes. Occasionally, you’ll need to SSH into those instances, either to troubleshoot an issue or to set something up that couldn’t be done from the Beanstalk console. If you’ve ever SSHed into a normal EC2 instance, you know you needContinue reading “SSH Into AWS Elastic Beanstalk Instances”

By default, Lambda runs your functions in a secure VPC with access to AWS services and the internet. The VPC is owned by Lambda and does not connect to your account’s default VPC. When you connect a function to a VPC in your account, it does not have access to the internet unless your VPCContinue reading “Lambda Functions in a VPC Timeout When Trying to Invoke Other Lambda Functions”

If using Lambda proxy integration, you can directly return the proper HTTP status codes & headers from your Lambda function: If not using Lambda proxy integration, create response mappings in API Gateway as shown below. First, add all HTTP status codes you intend to use in Method Response: Next, in Integration Response, map each HTTPContinue reading “Define Custom HTTP Status Codes to be Returned by Amazon API Gateway”

The easiest way to set environment variables on Beanstalk apps is from the console: But if you need your variables as a part of your deployment package, maybe because you want them under version control or because you want they to be applied to any new environments you create, simple create a .ebextensions/*.config file: AfterContinue reading “How to Set Custom Environment Variables on an AWS Elastic Beanstalk App?”

To run a cron task periodically, create a folder at the root of your application called .ebextensions. Then create a config file inside the .ebextensions folder with this: leader_only key ensures the command is only run on the EC2 instance that is considered the leader & not on every instance you have running. Put yourContinue reading “How to Run a Cron Job in AWS Elastic Beanstalk?”

If you occasionally find that your EC2 instance runs out of memory & you don’t wanna upgrade to a larger instance, consider adding swap space (a.k.a. paging) to it! Paging works by creating an area on your hard drive & using it for extra memory. This is much slower than normal memory but a lotContinue reading “Add Swap Space to EC2 Instances to Avoid Having to Upgrade to a Larger Instance”

So you moved your domain to Route 53, created a hosted zone with all the right records & yet when you visit your domain every after days after this change, you see the error “dial tcp: lookup example.com on 8.8.8.8:53: server misbehaving” & when you try DNS lookup, it says it does not have anyContinue reading ““dial tcp: lookup example.com on 8.8.8.8:53: server misbehaving” After Transferring Your Domain Over to Amazon Route 53″

Whether you have a webapp deployed in Beanstalk or a REST API, you might have come across the HTTP 413 error when trying to POST or upload a file larger than 1 MB to it. This is because the Nginx server running in Beanstalk is configured to accept no more that 1 MB of dataContinue reading “Uploading Large Files to AWS Elastic Beanstalk Fails with HTTP 413 “Request Entity Too Large””

What is the point of having private subnets at all when you can simply take away an EC2 instance’s public IP to make it private? The main difference between a public & private subnet is what the subnet’s default route is, in the VPC’s routing tables. This determines the validity of public IPs on instancesContinue reading “Why Use Private Subnets in a VPC When Instances Without a Public IP in Public Subnets are Private Anyway?”

If you’ve deployed a static website on an S3 bucket behind a CloudFront distribution & encountered the following error when you navigate the site, this post is for you. Weirdly, this will work if you hit the same object’s S3 URL directly, but not if you go via CloudFront. So what’s going on here? Let’sContinue reading ““The specified key does not exist” for Static Websites Hosted on S3 behind CloudFront”

Every EC2 instance has associated metadata, which AWS makes available to all users & applications inside the instance. The instance ID is part of this metadata. Here’s a complete list of everything included in the metadata — Instance metadata categories. Run the following at a Bash prompt to get the instance ID: The same canContinue reading “How to Get the EC2 Instance ID from Within an EC2 Instance?”

There are many ways for a Lambda function to invoke another Lambda function. The first is to use the AWS SDK directly. JavaScript Here’s an example using the JavaScript SDK: See Lambda.invoke’s documentation here. Python The same can be done using Boto 3: That’s async invocation. For sync, do this: Java Here’s a detailed blogContinue reading “Lambda Calling Lambda”

If you use a public S3 bucket to serve up static assets like images, fonts, etc. to your websites, you must have encountered the infamous CORS error. CORS stands for Cross-Origin Resource Sharing. The error looks something like this in Chrome console: S3 provides a simple way to enable CORS. Simply add the following CORSContinue reading “Access-Control-Allow-Origin — Enable CORS for Static Assets in S3”

It’s common for APIs to have parameters, either as query strings — GET /object?name=book — or as path parameters — GET /object/book. But if your APIs are implemented in Lambda functions, how do you access the values of these parameters in a Lambda function? Here are a few ways: Using Lambda Proxy Integration Turn onContinue reading “Access Query Strings, Path Parameters & HTTP Headers in Lambda Functions Behind Amazon API Gateway”

If you’re using S3’s static website hosting feature to host a React or Angular webapp, the URLs in the URL bar of a browser running your webapp are essentially pseudo URLs. There’s no HTML page at the end of any of those URLs. Client-side JavaScript in your webapp handles all URLs & displays the appropriateContinue reading “Redirect All Routes to index.html When Hosting a React/Angular Webapp on S3”

Say you have a webapp that lets users upload a file to S3, immediately followed by a download request for the same file. A user’s profile picture is a common example. As soon as the user has uploaded their profile picture, you have to download it to show it on their profile page. If theContinue reading “CloudFront + S3 Behavior in Download After Upload Scenarios”

AWS4-HMAC-SHA256, also known as Signature Version 4 or Sig v4 is 1 of 2 authentication mechanisms supported by S3. All AWS regions support Sig v4 but N. Virginia (US East 1) & many others also support the older Sig v2. Amazon S3 supports Signature Version 4, a protocol for authenticating inbound API requests to AWSContinue reading “The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.”

Whether you’re using “microservices in a monorepo” design or just need a subdirectory of a large GitHub repo, you definitely don’t have to clone the entire repo to work with just a handful of files. Instead, use GitHub’s ability to convert any to repo to SVN, to get just the files you need.

Lambda functions can be triggered by a number of sources by creating event source mappings, but if you need to enable/disable these mappings programmatically, to control when the Lambda function triggers, the most straightforward way is to call Lambda’s UpdateEventSourceMapping API. But if for some reason, you need your own API wrapper over this API, API Gateway is the way to go. This post explains how to set up an API in API Gateway to enable/disable your Lambda triggers.

Lambda environment variables are encrypted at rest by default but anyone with access to the Lambda console of your AWS account can see the values of your environment variables. This is fine for variables like database endpoints but not good for variables that store some kind of secret, like a database password for example. This post shows you how to obfuscate & protect them.

SolarWinds Loggly is a log aggregation & analysis platform. You can send logs to Loggly from many sources. This article describes the steps to send CloudWatch logs to Loggly.

This article summarizes the Load Balancing service in Oracle Cloud Infrastructure.

This article summarizes the Container Engine for Kubernetes in Oracle Cloud Infrastructure, also known as OCI OKE.

Amazon API Gateway provides usage plans with quotas that can be used to limit the number of requests to your API. Quotas are enforced on a per-customer basis. This is the proper design for most real-world API projects. But what if you’re building small-scale projects or personal projects where the cost incurred to you by your API in your AWS account is the biggest concern. What if you have a hard spending limit for API Gateway in your AWS account & you’re OK with shutting down the API for everyone as soon as the spending limit is reached. This post shows you how to do just that.

Amazon API Gateway supports usage plans which contain quotas to limit the number of requests a user can make to your API in a given time period. Quotas work by inspecting the HTTP header carrying the API key in every incoming request. But what about requests that do not or cannot support specifying API keys? API Gateway cannot enforce any quotas on them. One such request is the HTTP OPTIONS method used by browsers in preflight requests to determine the CORS status of API endpoints, before making the actual call to the API. This article describes a way to use WAF to limit OPTIONS calls.

Although the primary use case of Amazon EFS is to be used as an NFS, occasionally it could hold the kind of data that lends well to version control. A WordPress installation is one such example. And where there is version control, automated deployment pipelines aren’t far behind. This article describes a way for you to automatically deploy files from a CodeCommit repository to an EFS file system, whenever a change is pushed to the repo.

This article summarizes the Events service in Oracle Cloud Infrastructure.

This article summarizes the Functions service in Oracle Cloud Infrastructure.

It’s fairly common in enterprises to have multiple AWS accounts for different environments: dev/test, staging/QA, prod/preprod, etc. Often there’s a shared services account which hosts all services that other accounts need. One such service is sending emails. This article explains how to set up SES in the shared account & use it to send emails from a Lambda function running in the dev or prod account.

WordPress is awesome! It powers 37% of the internet & is the most beloved CMS out there. But like everything else, it can always be improved. If you’re using WordPress to serve up a simple website with occasionally changing content, then you can benefit a lot by turning it into a static site. You don’t have to get rid of WordPress though. Keep using it for the ease of content generation, but convert everything into plain-old HTML & CSS before serving it to the world. Here’s how to do that.

This post summarizes the API Gateway service in Oracle Cloud Infrastructure.

If you spend a lot of time managing cloud infrastructure manually via the cloud provider’s web console, you often need to come up with unique names for your cloud resources. You probably have a naming convention in place for most cloud services that you work with, but when it comes to global services like Amazon S3, you must resort to unique alpha-numeric “generated” identifier strings for at least a portion of the resource name. This article provides a one-click generate & copy solution for such names. All you have to do is click & paste!

It’s a common practice in traditional applications to maintain a pool of ready-to-use database connections, instead of creating one from scratch every time a piece of code needs to talk to the database. But what happens if you do the same in Lambda functions? Each function is separate from the other & cannot share the pool. How do connection pools behave then? Are they even required? How should you size them? This article aims to explore these questions & explain in detail how database connection pools behave in Lambda functions.

AWS Lambda doesn’t allow the deployment package of a Lambda function to be larger than 50 MB in size. But that’s only when you upload the package directly to the function. If instead, you upload the package to S3 & provide the S3 URL to the Lambda function, you can get away with deployment packages up to 250 MB in size (uncompressed)!

Say you have a bunch of EC2 instances in an auto scaling group & you have a requirement to be able to connect to them via SFTP. This post shows you how to do just that!

Over time, as the data in your production database grows, it will inevitably degrade the DB’s performance. When that happens, you know it’s time to get rid of some of that years old data. But of course, you can’t just delete it. You have to archive it first. This article describes a cloud-native way of doing this.

Say you have 2 AWS accounts, 1 for dev & 1 for prod. Now in most cases, devs have no access to prod & that’s how it should be. But when things break in prod, it would help to let the devs take a look in there. This article explains how to grant devs temporary access to prod & revoke it right after.

This tutorial will walk you through how to set up your development environment, then create & deploy a simple .NET Core 3.1 Lambda function using just the .NET Core CLI.

This post teaches you how to use CloudWatch Logs Insights’ built-in filtering, parsing & grouping capabilities to analyze logs.

This article summarizes OCI’s compute service: VMs, BMs, instance shapes, storage, best practices, data protection & more!

This post describes how to setup HTTPS using a self-signed certificate for a Python 3.7 webapp deployed to a single-instance AWS Elastic Beanstalk environment without using a custom domain or a load balancer. This is useful in dev/test scenarios where HTTPS is required.

This article shows you how to set up a fully-automated CI/CD pipeline for your AWS SAM application.

This article describes the steps involved in creating a simple hello world AWS SAM application in Node.js 12 & deploying it to AWS using the AWS toolkit in VS code.

In this article, you’ll see how easy it is to create a serverless API backend using AWS Lambda functions & Amazon API Gateway, backed by the NoSQL DynamoDB database, by deploying a sample Lambda application. This sample also comes with a CodeBuild project & a CodePipeline to fully automate deployments. It’ll even create a Git repository for you in CodeCommit so you can simply clone the repo & start building on top of the sample.

This post describes how to setup HTTPS using a self-signed certificate for a Node.js 12 webapp deployed to a single-instance AWS Elastic Beanstalk environment without using a custom domain or a load balancer. This is useful in dev/test scenarios where HTTPS is required.

This post covers every aspect of networking in OCI.

S3 costs increase linearly as the number of users downloading objects increases. This can make it expensive to distribute popular objects. BitTorrent addresses this problem.

This post describes how Amazon Elastic File System can be mounted inside Lambda functions & shared among Lambda, EC2 & on-prem.

This post describes how to run AWS CLI inside a Lambda function in a few easy steps.

This post describes how to run bash commands & scripts in a Lambda function easily.

This post describes how you can use EC2 user data to auto-assign static private IPs to instances in an autoscaling group.

This post describes how to use Amazon CloudWatch Logs metric filters to extract custom metrics from application logs.

This post teaches you how to deploy an Angular app on S3.

This post summarizes everything you need to know about NoSQL Database in Oracle Cloud Infrastructure.

This post summarizes everything you need to know about Traffic Management in Oracle Cloud Infrastructure.

This article summarizes everything you need to know about the DNS service in OCI.

MySQL Workbench has an in-built ability to connect to databases via an SSH tunnel. This feature can be used to connect to private RDS databases by establishing an SSH tunnel through the bastion host.

This article explains how you can use Secure Pipes on Mac to setup local port forwarding via an SSH tunnel. Secure Pipes can also setup remote port forwarding in the same way.

This article explains how Chrome DevTools can be used to take screenshot of a single HTML DOM node, instead of the entire screen or page.

This article describes how you can restrict the use of certain OAuth scopes to certain Okta apps by adding access policies with access rules to the authorization server.

This article explains how to embed Git credentials in the repo URL & avoid entering them everytime.

This article explains how to use an S3 bucket as an SFTP server.

This article explains how to create & deploy an AWS State Machine with a .NET Core Lambda using AWS SAM.