====== DaVinci Resolve in the cloud ======
I'm trying to keep up with my notes on [[:video_editing|Video Editing]] and [[video editing:DaVinci Resolve]]. Hardware choices for DaVinci resolve is a subject for a small book (which needs updating every 6 months), but I've summarised my notes under [[Video Editing:DaVinci Resolve Specs]].
===== Summary =====
My questions were:
> Can I use cloud computing to run DaVinci Resolve?
> At what point does it become cheaper to buy a new laptop for "editing on the go" in comparison to using cloud computing?
This came from the fact that I had an older (but not that old) laptop which could no-longer keep up with running DaVinci resolve (at least reasonably). I didn't need to upgrade it for any productivity outside of running DaVinci Resolve.
Laptop prices showed that a laptop with an Nvidia card that would run DaVinci Resolve confidently for a few years would cost about £3,000 If I went with Apple, then the new M1 chip brings the costs down. The Macbook Air which comes in at just over £1,000 (Or just under £1,500 for a more powerful Macbook Pro with the M1 chip).
After doing some numbers (just for DaVinci Resolve), I've calculated (at least for UK prices) that a £1,000 laptop used 10 days a year over a 5 year lifetime works out at £20 a day. Worst case scenario is a £3,000 laptop used for 5 days a year over a 3 year lifetime, which averages out at £200 a day. That's a lot of variance!
> If I can find a way of working which comes to less than £20 ($27.36) a day then I'm on par with my cheapest solution for my predicted usage.
This can be achieved by using one of:
* ''g4dn.2xlarge'' - [[https://aws.amazon.com/marketplace/pp/prodview-xrrke4dwueqv6|NVIDIA Gaming PC - Windows Server 2019]]
* ''g4dn.4xlarge'' - [[https://aws.amazon.com/marketplace/pp/prodview-zzy5tef4cq6sg|DaVinci Resolve 17 Studio]]
===== Note =====
This looks at using powerful windows virtual machines in the cloud, they will run both versions of DaVinci Resolve (Free and Studio). However studio will be able to use more of the fancy GPU features. Warning from personal experience, getting a dongle to work remotely is hard/impossible (I couldn't do it) and I've been unable to test with a licence key (so much for the dongle for when I'm working without internet).
This mostly looks at two EC2 instances:
* [[https://aws.amazon.com/marketplace/pp/prodview-xrrke4dwueqv6|NVIDIA Gaming PC - Windows Server 2019]]
* [[https://aws.amazon.com/marketplace/pp/prodview-zzy5tef4cq6sg|DaVinci Resolve 17 Studio]] - New as of June 2021 (I couldn't find much extra information on this, [[https://web.archive.org/web/*/https://aws.amazon.com/marketplace/pp/prodview-zzy5tef4cq6sg|WayBackMachine]] first found the page on June 8th 2021).
===== Scenario =====
I work from a PC which has good hardware, and then I have a laptop (which is now getting old) which I use on the go and for work related stuff. The laptop is capable of doing most things I require with the exception that it runs resolve painfully slowly!
Do I:
* Buy a new laptop with the hardware to run Resolve happily for a few years... In the windows world we are easily talking about £2k or more, more than I want to spend on something I don't use that often.
* Buy a new M1 chip laptop at around £1k, it's tempting but I don't want to invest so much in hardware.
* Put up with a laptop which is just frustrating with its workflow.
* Something else...
I don't tend to use the laptop except on some occasions when I'm on the go, so perhaps 5 days a year worth of time (under COVID we can bump that up to 10 days). For hardware, I'm going to expect it to be happy for 3 years, and usable for 5.
==== Cost Breakdown ====
This is what different costs look like. The lowest cost option is if I spend only £1,000 on a laptop and that laptop keeps me happy for 5 years (from 2021 to 2026), and I use that laptop 10 times a year for video editing (yes, I appreciate there will be other advantages to it). But that still lands me at £20 a day and if I actually go to only 5 days a year of usage, that's £40. Switching to only being happy with it for 3 years, that's £33.33 and £66.67.
^ Cost ^ Years ^ Cost per year ^ Days per year ^ Cost per day ^
| £1,000.00 | 5 | £200.00| 10 | £20.00|
| £2,000.00 | 5 | £400.00| 10 | £40.00|
| £3,000.00 | 5 | £600.00| 10 | £60.00|
| £1,000.00 | 3 | £333.33| 10 | £33.33|
| £2,000.00 | 3 | £666.67| 10 | £66.67|
| £3,000.00 | 3 | £1,000.00| 10 | £100.00|
| £1,000.00 | 5 | £200.00| 5 | £40.00|
| £2,000.00 | 5 | £400.00| 5 | £80.00|
| £3,000.00 | 5 | £600.00| 5 | £120.00|
| £1,000.00 | 3 | £333.33| 5 | £66.67|
| £2,000.00 | 3 | £666.67| 5 | £133.33|
| £3,000.00 | 3 | £1,000.00| 5 | £200.00|
===== Enter The Cloud =====
I work with cloud computing professionally, as well as using [[https://m.do.co/c/bb281f2fae53|DigitalOcean]] for a virtual machine for my wiki, blog and other projects.
Note: The link above is my Referral link, If you sign up using it we both win! Everyone I refer gets $100 in credit over 60 days. Once they’ve spent $25 with DigitalOcean, I'll get $25 (we all win).
I found this article [[https://aws.amazon.com/blogs/media/creating-a-feature-film-with-davinci-resolve-on-aws/|Creating a feature film with DaVinci Resolve on AWS]] and was inspired to think differently. While the article focuses on providing a super powerful environment and infrastructure for supporting a large project, the same philosophy can apply for occasional hobby usage.
I also found an article and PDF on using AWS environments for doing 3D rendering and graphics work:
* [[https://studio-in-the-cloud-tutorials.s3-us-west-1.amazonaws.com/streaming-workstation/Creating+a+Virtual+Workstation+on+AWS.pdf|Creating a Linux Virtual Workstation on AWS]] - PDF
And an article for AWS and Post Production (Adobe CC) products:
* [[https://aws.amazon.com/blogs/media/running-adobe-creative-cloud-suite-on-aws/|Deploying Your Favorite Post Production Applications on AWS Virtual Desktop Infrastructure]]
==== Cost Breakdown ====
I calculated, that if I can find a way of working which comes to less than £20 a day then I'm on par with my cheapest solution for my predicted usage.
AWS is priced in USD, at the time of this note, [[https://www.xe.com/currencyconverter/convert/?Amount=10&From=GBP&To=USD|Xe Currency Converter]]: 10 GBP to USD = 13.6823 US Dollars. So my aim is to stay below $27.36 USD per day.
To compare, I looked at the EC2 prices based on the Paris region:
[[https://aws.amazon.com/marketplace/pp/prodview-xrrke4dwueqv6|NVIDIA Gaming PC - Windows Server 2019]]:
^ Model| g4dn.xlarge | g4dn.2xlarge | g4dn.4xlarge |
^ Cores| 4 | 8 | 16 |
^ Memory (GB)| 16 | 32 | 64 |
^ Hour Cost (USD)| 0.799 | 1.247 | 2.144 |
^ 8 Hour Day (USD)| 6.392 | 9.976 | 17.152 |
^ 10 Hour Day (USD)| 7.99 | 12.47 | 21.44 |
^ 12 Hour Day (USD)| 9.588 | 14.964 | 25.728 |
Using the 16 core machine for 12 hours (marathon day), I'm keeping (just) below the $27.36/£20 a day price I'm aiming for, although tax and data storage might push it over. So, instead to remain cost effective I'm better either using it for a shorter period of time or dropping down to a smaller machine. Fortunately using cloud resources makes this scaling up and down easy.
[[https://aws.amazon.com/marketplace/pp/prodview-zzy5tef4cq6sg|DaVinci Resolve 17 Studio]]:
^ Model| g4dn.xlarge | g4dn.2xlarge | g4dn.4xlarge | g4dn.8xlarge |
^ Cores| N/A | N/A | 16 | 32 |
^ Memory (GB)| N/A | N/A | 64 | 128 |
^ Hour Cost (USD)| N/A | N/A | 1.409 | 2.546 |
^ 8 Hour Day (USD)| N/A | N/A | 11.272 | 20.368 |
^ 10 Hour Day (USD)| N/A | N/A | 14.09 | 25.46 |
^ 12 Hour Day (USD)| N/A | N/A | 16.908 | 30.552 |
Using the 16 core machine for 12 hours (marathon day), I'm keeping well below the $27.36/£20 a day price I'm aiming for. I could go up to the more powerful instance for shorter bursts of productivity, if the requirement was there.
When you stop (turn off) the virtual machine, you stop paying the prices above. Instead you only pay for the storage:
> When you stop an instance, we shut it down. We don't charge usage for a stopped instance, or data transfer fees, but we do charge for the storage for any Amazon EBS volumes.
* https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Stop_Start.html
At time of writing, the [[https://aws.amazon.com/ebs/pricing/|costs for EBS]] show:
gp2 - 30 GB = $ 3.48
gp3 - 200 GB = $ 18.56
By default, the instance configuration contains the 30BG of gp2 (plus a 200GB disk which is thrown out when the machine is off), so it will cost a total of $3.48 for a month of a switched off machine (software installed and configured).
If you want to store more data for longer periods (like over the month) it's better to use S3 (checked against the [[https://aws.amazon.com/s3/pricing/|s3 pricing]]).
^ Type:^ Cost per GB-month ^ GB ^ Monthly Cost ^
^ General Purpose SSD (gp2) Volumes| $0.1160 | 30 | $3.48 |
^ General Purpose SSD (gp3) – Storage| $0.0928 | 30 | $2.78 |
^ General Purpose SSD (gp3) – Storage| $0.0928 | 200 | $18.56 |
^ S3 Standard - First 50 TB / Month| $0.0240 | 200 | $4.80 |
^ S3 Standard - First 50 TB / Month| $0.0240 | 1,000 | $24.00 |
===== Setup AWS For DaVinci Resolve =====
I'm going to focus on two key areas of the AWS catalogue (it's massive, they have offerings and options than the biggest supermarket has breakfast cereals).
- **EC2** (Elastic Compute Cloud): This is a virtual machine offering, allowing people to create as many virtual computers in the cloud as they like within minuets. Then thrown them away when they have finished with them, only paying for the time that the machine "exists". [[https://aws.amazon.com/ec2/|Amazon EC2 - Amazon AWS]]
- **S3** (Simple Storage Service): This is virtual disk storage, hosted by amazon. It's commonly used for big data solutions, allowing users to create "buckets" to put data into. But it's also a cost effective way of pushing small to extremely large amounts of data into the cloud and into the AWS ecosystem (ready for use by an EC2 instance. [[https://aws.amazon.com/s3/|Simple Storage Service (Amazon S3) - Amazon AWS]]
EC2 is the main focus of this workflow, as I'm interested in leveraging the available [[https://aws.amazon.com/nvidia/|AWS and NVIDIA]] virtual machines. DaVinci Resolve leverages the GPU, and having a nice powerful [[https://aws.amazon.com/ec2/instance-types/g4/|NVIDIA T4 GPU]]. Note, only DaVinci Resolve Studio (non free version) will fully utilise the GPU (although the free version gets a good portion of it).
==== AWS Account ====
I can't be bothered to cover this, it's easy to sign up for an account with amazon for AWS and get that up and running (do allow a little time for it).
==== AWS Configuration ====
To be able to run the G4dn Instances under EC2, I had to extend my quota: [[https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-resource-limits.html|Amazon EC2 service quotas]]
> To request an increase using the Amazon EC2 console
> - Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
> - From the navigation bar, select a Region.
> - From the navigation pane, choose Limits.
> - Select the resource in the list, and choose Request limit increase.
> - Complete the required fields on the limit increase form and choose Submit. We'll respond to you using the contact method that you specified.
Configure Service Quota:
* ''Running On-Demand G instances'' - I set this to 16 so I could use up to (but not accidentally more than) a 16 core machine.
===== AWS DaVinci Resolve Workflow =====
==== Data Upload (optional) ====
=== Configure Access Tokens ===
Before bucket, you will need these in VM. Easy/command line access to S3 bucket.
Tokens:
* https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys
=== Create Bucket ===
Follow S3 instructions
=== Upload Data ===
Follow S3 instructions
==== Create VM ====
You will want to create one of these, it's a preconfigured server image for the Nvidia hardware, you can select the number of vCPU cores and amount of RAM:
* [[https://aws.amazon.com/marketplace/pp/B07STLTHM8|NVIDIA Gaming PC - Windows Server 2019]]
Select VM type, I recommend filtering for g4dn and selecting either the 8 or 16 core machine (more money = more power).
You will need to get the admin password (wait 4 mins after creation of VM), that required using the key pair.
==== Connect To VM ====
Once you have the hostname, username and password, you can connect using Windows Remote Desktop (search for ''mstsc'' in the start menu)
Note: It does allow you to download a configuration file with the username and hostname which you can use to launch the application rather than copy and paste into the application.
If you are reconnecting:
- Check the firewall rules, by default the setup uses the IP address from which you first connected. So, if you connect from a new outside location (or your home IP changes) the firewall will block you.
- Check the Public DNS. If you stop, then restart the EC2 instance, the IP address (and Public DNS associated with it) will change.
If you get too many old connections:
* https://docs.microsoft.com/en-us/troubleshoot/windows-server/remote/remove-entries-from-remote-desktop-connection-computer
* Remove entries from: ''HKEY_CURRENT_USER\Software\Microsoft\Terminal Server Client\Default''
==== Configure VM & Install Software ====
DaVinci Resolve fails to install (thanks to the workings of default PostgreSQL installation). As this isn't a production world facing server, we can cut back the security.
- Local Security Policy (Editor)
- Account Policies -> Password Policy: Password must meet complexity requirements = Disabled
- Local Group Policy Editor
- Computer Configuration -> Administrative Templates -> Windows Components -> Internet Explorer -> Internet Control Panel -> Advanced Page -> Turn On Enhanced Protected Mode = Disabled
Other Configuration steps
- Format/create hard disk
- Note: If you "stop" the VM to reconnect again, the disk D will be gone
- Enable "Windows Audio" service (set to start on startup)
Software installation:
- AWS CLI (for S3)
- https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-windows.html
- https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html
- ninite.com Notepad++, peazip, VLC
- Download resolve: https://www.blackmagicdesign.com/products/davinciresolve/
Optional Session Manager
- https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-install-ssm-win.html
- https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-install-win.html
- https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-install-win.html
==== Data Import ====
Note: If you "stop" the VM to reconnect again, the disk D will be gone
Copy vir RDP...
or, using S3
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-instance-access-s3-bucket/
AWS cli, copy files to/from S3
* https://aws.amazon.com/cli/
* https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html
* https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonS3.html
* https://www.thegeekstuff.com/2019/04/aws-s3-cli-examples/
Using access keys from above
aws configure
aws s3 ls s3://
aws s3 ls s3://drh-video-demo-1
D:\>aws s3 cp s3:/// D:\ --recursive
D:\>aws s3 cp s3://drh-video-demo-1/ D:\ --recursive
==== Get Going ====
You should now be able to use the VM via remote desktop like any normal machine.
==== Backup & Export ====
D:\>aws s3 cp s3:/// --recursive
D:\>aws s3 cp dump s3://drh-video-demo-1/d1 --recursive
==== Terminate VM ====
This is super important as you pay for the machine all the time you are using it.
===== Other Notes =====
* [[https://aws.amazon.com/marketplace/pp/B07SSKJMBJ|NVIDIA Gaming PC - Ubuntu 18.04]] - Linux machine which could be made to work with resolve... I haven't tested it but the running cost of a Linux VM is lower than the PC version. It has limitations on supported media formats (outside of studio version).
* [[https://github.com/aws-samples/aws-cloud-video-editing|aws-cloud-video-editing]] - Scripts to automate configuration of S3 & Cloud video environments, I haven't tested these.
* [[https://aws.amazon.com/blogs/media/tag/teradici/|teradici - AWS Blog Posts]]
* https://teradici.com/ - Teradici provide a high performance and more advanced virtual desktop solution. Note that this costs an extra 0.50 USD per hour for EC2.
* [[https://www.youtube.com/watch?v=UuylupouVFI|Setup Davinci Resolve Project Server on the Amazon Cloud - A cloud tutorial]]
* Dongle (Can't get dongle to work):
* https://bagel-labs.zendesk.com/hc/en-us/articles/360019016534-Using-the-USB-Dongle-with-Windows-Remote-Desktop
* https://workspot.zendesk.com/hc/en-us/articles/214248563-Configuring-USB-Redirection-with-RemoteFX-in-Workspot
====== AWS DaVinci Resolve Workflow Reconnect ======
===== Upload new files =====
Use a bat script to opy the files from local machine to s3.
* https://superuser.com/questions/444726/windows-how-to-add-batch-script-action-to-right-click-menu
@ECHO OFF
REM ===========================================================================
REM Set common variables
REM project_list will be whatever directories you wish to copy from the s3 projects directory to local
REM ===========================================================================
SET aws_s3_bucket=drh-video1
SET drive_location="C:\Users\david\Pictures\Adventures In Cloud Computing"
SET project_list=dir_name_1 dir_name_2
REM remove quotes
SET drive_location=%drive_location:"=%
REM ===========================================================================
REM Set varialbes used in script
REM ===========================================================================
REM SET local_project_location=%drive_location%\projects
SET local_project_location=%drive_location%
SET s3_project_location=s3://%aws_s3_bucket%/projects
echo Local project location: %local_project_location%
echo S3 project location: %s3_project_location%
REM ===========================================================================
REM Push projects up
REM ===========================================================================
FOR %%i in (%project_list%) do (
echo Project: %%i
aws s3 sync "%local_project_location%/%%i" %s3_project_location%/%%i
)
pause
Remember before connecting:
- Check new IP/DNS Name
===== Create D drive =====
The second disk isn't kept, but can be recreated using a script
After connecting:
- Format/create hard disk: [[https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/diskpart-scripts-and-examples]]
select disk 1
clean
create partition primary
format quick
assign letter=D
diskpart /s script_diskpart.txt
pause
===== Copy files from projects in s3 =====
@ECHO OFF
REM ===========================================================================
REM Set common variables
REM project_list will be whatever directories you wish to copy from the s3 projects directory to local
REM ===========================================================================
SET aws_s3_bucket=drh-video1
SET drive_location=D:\projects
REM SET drive_location=C:\Users\david\projects
SET project_list=OVFM logo
REM ===========================================================================
REM Set varialbes used in script
REM ===========================================================================
SET local_project_location=%drive_location%
SET s3_project_location=s3://%aws_s3_bucket%/projects
echo Local project location: %local_project_location%
echo S3 project location: %s3_project_location%
REM ===========================================================================
REM List available projects
REM ===========================================================================
echo ===========================================================================
echo Available Projects
echo ===========================================================================
aws s3 ls %s3_project_location%/
REM ===========================================================================
REM Create some local directories
REM ===========================================================================
mkdir %drive_location%
mkdir D:\MEDIA
REM ===========================================================================
REM Pull projects down
REM ===========================================================================
echo ===========================================================================
echo Downloading Configured Projects
echo ===========================================================================
FOR %%i in (%project_list%) do (
aws s3 cp %s3_project_location%/%%i %local_project_location%\%%i --recursive
)
pause
===== Copy files from projects in D to s3 =====
@ECHO OFF
SETLOCAL ENABLEDELAYEDEXPANSION
REM ===========================================================================
REM Set common variables
REM ===========================================================================
SET aws_s3_bucket=drh-video1
SET drive_location=D:\projects
REM ===========================================================================
REM Set varialbes used in script
REM ===========================================================================
SET local_project_location=%drive_location%
SET s3_project_location=s3://%aws_s3_bucket%/projects
echo Local project location: %local_project_location%
echo S3 project location: %s3_project_location%
REM ===========================================================================
REM Loop directories in projects directory and sync to s3
REM ===========================================================================
for /d %%I in (%local_project_location%\*) do (
SET FullDirName=%%I
SET CurrDirName=%%~nxI
echo Project Name: !CurrDirName!
echo Project Path: !FullDirName!
aws s3 sync !FullDirName! %s3_project_location%/!CurrDirName!
)
pause