GPU ML Pipeline on AWS with Terraform | From infrastructure code to GPU-accelerated ML jobs at scale
I built this Terraform-managed AWS ML pipeline to run GPU and CPU jobs on demand without paying for idle compute. In this post, I walk through the architecture, the key design choices, and the operational pitfalls to watch for before using this pattern in production.
<details id="toc-wrapper" style="border: 1px solid var(--primary-color, #444444); padding: 0.75rem 1rem; margin: 1rem 0; border-radius: 0.25em;">
<summary style="cursor: pointer; font-weight: 600;">Table of Contents</summary>
<nav id="TOC" role="doc-toc" style="margin-top: 0.75rem; text-align: left;">
<ul>
<li><a href="#introduction" id="toc-introduction"><span class="toc-section-number">1</span> Introduction</a></li>
<li><a href="#architecture" id="toc-architecture"><span class="toc-section-number">2</span> Architecture</a></li>
<li><a href="#trigger-flow" id="toc-trigger-flow"><span class="toc-section-number">3</span> Trigger Flow</a></li>
<li><a href="#compute" id="toc-compute"><span class="toc-section-number">4</span> Compute</a></li>
<li><a href="#docker-images" id="toc-docker-images"><span class="toc-section-number">5</span> Docker Images</a></li>
<li><a href="#job-scripts" id="toc-job-scripts"><span class="toc-section-number">6</span> Job Scripts</a></li>
<li><a href="#admin-tools" id="toc-admin-tools"><span class="toc-section-number">7</span> Admin Tools</a></li>
<li><a href="#state-and-backend" id="toc-state-and-backend"><span class="toc-section-number">8</span> State and Backend</a></li>
<li><a href="#gpu-quotas" id="toc-gpu-quotas"><span class="toc-section-number">9</span> GPU Quotas</a></li>
<li><a href="#code" id="toc-code"><span class="toc-section-number">10</span> Code</a></li>
</ul>
</nav>
</details>
<h1 data-number="1" id="introduction"><span class="header-section-number">1</span> Introduction</h1>
<p>The project is a Terraform-managed infrastructure for running GPU and CPU ML workloads on AWS. You publish a message to an SNS topic, a Lambda function routes it to AWS Batch, Batch spins up an EC2 instance, runs your code in a Docker container, uploads the outputs to S3, and shuts down. When there are no jobs, the compute environments sit at zero vCPUs and cost nothing.</p>
<h1 data-number="2" id="architecture"><span class="header-section-number">2</span> Architecture</h1>
<img width="100%" src="data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9Im5vIj8+CjwhRE9DVFlQRSBzdmcgUFVCTElDICItLy9XM0MvL0RURCBTVkcgMS4xLy9FTiIKICJodHRwOi8vd3d3LnczLm9yZy9HcmFwaGljcy9TVkcvMS4xL0RURC9zdmcxMS5kdGQiPgo8IS0tIEdlbmVyYXRlZCBieSBncmFwaHZpeiB2ZXJzaW9uIDIuNDMuMCAoMCkKIC0tPgo8IS0tIFRpdGxlOiBBV1NfTUxfUGlwZWxpbmUgUGFnZXM6IDEgLS0+Cjxzdmcgd2lkdGg9IjEwODBwdCIgaGVpZ2h0PSI2MTJwdCIKIHZpZXdCb3g9IjAuMDAgMC4wMCAxMDc5LjgwIDYxMS44MCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB4bWxuczp4bGluaz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94bGluayI+CjxnIGlkPSJncmFwaDAiIGNsYXNzPSJncmFwaCIgdHJhbnNmb3JtPSJzY2FsZSgxIDEpIHJvdGF0ZSgwKSB0cmFuc2xhdGUoMTQuNCA1OTcuNCkiPgo8dGl0bGU+QVdTX01MX1BpcGVsaW5lPC90aXRsZT4KPHBvbHlnb24gZmlsbD0id2hpdGUiIHN0cm9rZT0idHJhbnNwYXJlbnQiIHBvaW50cz0iLTE0LjQsMTQuNCAtMTQuNCwtNTk3LjQgMTA2NS40LC01OTcuNCAxMDY1LjQsMTQuNCAtMTQuNCwxNC40Ii8+CjxnIGlkPSJjbHVzdDEiIGNsYXNzPSJjbHVzdGVyIj4KPHRpdGxlPmNsdXN0ZXJfdGVycmFmb3JtPC90aXRsZT4KPHBhdGggZmlsbD0id2hpdGUiIHN0cm9rZT0iI2NiYTNiYyIgZD0iTTM1NSwtNDQ4QzM1NSwtNDQ4IDEwMzEsLTQ0OCAxMDMxLC00NDggMTAzNywtNDQ4IDEwNDMsLTQ1NCAxMDQzLC00NjAgMTA0MywtNDYwIDEwNDMsLTUwOCAxMDQzLC01MDggMTA0MywtNTE0IDEwMzcsLTUyMCAxMDMxLC01MjAgMTAzMSwtNTIwIDM1NSwtNTIwIDM1NSwtNTIwIDM0OSwtNTIwIDM0MywtNTE0IDM0MywtNTA4IDM0MywtNTA4IDM0MywtNDYwIDM0MywtNDYwIDM0MywtNDU0IDM0OSwtNDQ4IDM1NSwtNDQ4Ii8+Cjx0ZXh0IHRleHQtYW5jaG9yPSJtaWRkbGUiIHg9IjY5MyIgeT0iLTUwNy4yIiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYiIGZvbnQtc2l6ZT0iMTEuMDAiPlRlcnJhZm9ybSBNYW5hZ2VkIEluZnJhc3RydWN0dXJlPC90ZXh0Pgo8L2c+CjwhLS0gdXNlciAtLT4KPGcgaWQ9Im5vZGUxIiBjbGFzcz0ibm9kZSI+Cjx0aXRsZT51c2VyPC90aXRsZT4KPHBhdGggZmlsbD0iI2U5ZjNmZiIgc3Ryb2tlPSIjMjIyMjIyIiBkPSJNMzE0LC01ODNDMzE0LC01ODMgMjM4LC01ODMgMjM4LC01ODMgMjMyLC01ODMgMjI2LC01NzcgMjI2LC01NzEgMjI2LC01NzEgMjI2LC01NTkgMjI2LC01NTkgMjI2LC01NTMgMjMyLC01NDcgMjM4LC01NDcgMjM4LC01NDcgMzE0LC01NDcgMzE0LC01NDcgMzIwLC01NDcgMzI2LC01NTMgMzI2LC01NTkgMzI2LC01NTkgMzI2LC01NzEgMzI2LC01NzEgMzI2LC01NzcgMzIwLC01ODMgMzE0LC01ODMiLz4KPHRleHQgdGV4dC1hbmNob3I9Im1pZGRsZSIgeD0iMjc2IiB5PSItNTYyLjUiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1zaXplPSIxMC4wMCI+VXNlciBvciBBZG1pbiBVSTwvdGV4dD4KPC9nPgo8IS0tIHRyaWdfdG9waWMgLS0+CjxnIGlkPSJub2RlMiIgY2xhc3M9Im5vZGUiPgo8dGl0bGU+dHJpZ190b3BpYzwvdGl0bGU+CjxwYXRoIGZpbGw9IiNlOWYzZmYiIHN0cm9rZT0iIzIyMjIyMiIgZD0iTTMxNCwtNDkyQzMxNCwtNDkyIDIzOCwtNDkyIDIzOCwtNDkyIDIzMiwtNDkyIDIyNiwtNDg2IDIyNiwtNDgwIDIyNiwtNDgwIDIyNiwtNDY4IDIyNiwtNDY4IDIyNiwtNDYyIDIzMiwtNDU2IDIzOCwtNDU2IDIzOCwtNDU2IDMxNCwtNDU2IDMxNCwtNDU2IDMyMCwtNDU2IDMyNiwtNDYyIDMyNiwtNDY4IDMyNiwtNDY4IDMyNiwtNDgwIDMyNiwtNDgwIDMyNiwtNDg2IDMyMCwtNDkyIDMxNCwtNDkyIi8+Cjx0ZXh0IHRleHQtYW5jaG9yPSJtaWRkbGUiIHg9IjI3NiIgeT0iLTQ3MS41IiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYiIGZvbnQtc2l6ZT0iMTAuMDAiPlNOUyBUcmlnZ2VyIFRvcGljPC90ZXh0Pgo8L2c+CjwhLS0gdXNlciYjNDU7Jmd0O3RyaWdfdG9waWMgLS0+CjxnIGlkPSJlZGdlMSIgY2xhc3M9ImVkZ2UiPgo8dGl0bGU+dXNlciYjNDU7Jmd0O3RyaWdfdG9waWM8L3RpdGxlPgo8cGF0aCBmaWxsPSJub25lIiBzdHJva2U9IiM0NDQ0NDQiIGQ9Ik0yNzYsLTU0Ni44NEMyNzYsLTUzMy4zIDI3NiwtNTE0LjI0IDI3NiwtNDk5LjE1Ii8+Cjxwb2x5Z29uIGZpbGw9IiM0NDQ0NDQiIHN0cm9rZT0iIzQ0NDQ0NCIgcG9pbnRzPSIyNzguNDUsLTQ5OS4xMSAyNzYsLTQ5Mi4xMSAyNzMuNTUsLTQ5OS4xMSAyNzguNDUsLTQ5OS4xMSIvPgo8L2c+CjwhLS0gZGlzcGF0Y2hlciAtLT4KPGcgaWQ9Im5vZGUzIiBjbGFzcz0ibm9kZSI+Cjx0aXRsZT5kaXNwYXRjaGVyPC90aXRsZT4KPHBhdGggZmlsbD0iI2U5ZjNmZiIgc3Ryb2tlPSIjMjIyMjIyIiBkPSJNNzMxLjUsLTQxOUM3MzEuNSwtNDE5IDY0NC41LC00MTkgNjQ0LjUsLTQxOSA2MzguNSwtNDE5IDYzMi41LC00MTMgNjMyLjUsLTQwNyA2MzIuNSwtNDA3IDYzMi41LC0zOTUgNjMyLjUsLTM5NSA2MzIuNSwtMzg5IDYzOC41LC0zODMgNjQ0LjUsLTM4MyA2NDQuNSwtMzgzIDczMS41LC0zODMgNzMxLjUsLTM4MyA3MzcuNSwtMzgzIDc0My41LC0zODkgNzQzLjUsLTM5NSA3NDMuNSwtMzk1IDc0My41LC00MDcgNzQzLjUsLTQwNyA3NDMuNSwtNDEzIDczNy41LC00MTkgNzMxLjUsLTQxOSIvPgo8dGV4dCB0ZXh0LWFuY2hvcj0ibWlkZGxlIiB4PSI2ODgiIHk9Ii0zOTguNSIgZm9udC1mYW1pbHk9IkhlbHZldGljYSxzYW5zLVNlcmlmIiBmb250LXNpemU9IjEwLjAwIj5MYW1iZGEgRGlzcGF0Y2hlcjwvdGV4dD4KPC9nPgo8IS0tIHRyaWdfdG9waWMmIzQ1OyZndDtkaXNwYXRjaGVyIC0tPgo8ZyBpZD0iZWRnZTIiIGNsYXNzPSJlZGdlIj4KPHRpdGxlPnRyaWdfdG9waWMmIzQ1OyZndDtkaXNwYXRjaGVyPC90aXRsZT4KPHBhdGggZmlsbD0ibm9uZSIgc3Ryb2tlPSIjNDQ0NDQ0IiBkPSJNMzEzLjIyLC00NTUuOTdDMzIwLjk2LC00NTIuOTQgMzI5LjE1LC00NTAuMDkgMzM3LC00NDggNDM1Ljc3LC00MjEuNzYgNTU0LjMzLC00MTAuMiA2MjUuMjksLTQwNS4zMiIvPgo8cG9seWdvbiBmaWxsPSIjNDQ0NDQ0IiBzdHJva2U9IiM0NDQ0NDQiIHBvaW50cz0iNjI1LjYxLC00MDcuNzYgNjMyLjQzLC00MDQuODQgNjI1LjI4LC00MDIuODcgNjI1LjYxLC00MDcuNzYiLz4KPC9nPgo8IS0tIGdwdV9xdWV1ZSAtLT4KPGcgaWQ9Im5vZGU0IiBjbGFzcz0ibm9kZSI+Cjx0aXRsZT5ncHVfcXVldWU8L3RpdGxlPgo8cGF0aCBmaWxsPSIjZWNmOWVmIiBzdHJva2U9IiMyMjIyMjIiIGQ9Ik03MjYuNSwtMzM3QzcyNi41LC0zMzcgNjQ3LjUsLTMzNyA2NDcuNSwtMzM3IDY0MS41LC0zMzcgNjM1LjUsLTMzMSA2MzUuNSwtMzI1IDYzNS41LC0zMjUgNjM1LjUsLTMxMyA2MzUuNSwtMzEzIDYzNS41LC0zMDcgNjQxLjUsLTMwMSA2NDcuNSwtMzAxIDY0Ny41LC0zMDEgNzI2LjUsLTMwMSA3MjYuNSwtMzAxIDczMi41LC0zMDEgNzM4LjUsLTMwNyA3MzguNSwtMzEzIDczOC41LC0zMTMgNzM4LjUsLTMyNSA3MzguNSwtMzI1IDczOC41LC0zMzEgNzMyLjUsLTMzNyA3MjYuNSwtMzM3Ii8+Cjx0ZXh0IHRleHQtYW5jaG9yPSJtaWRkbGUiIHg9IjY4NyIgeT0iLTMxNi41IiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYiIGZvbnQtc2l6ZT0iMTAuMDAiPkJhdGNoIEdQVSBRdWV1ZTwvdGV4dD4KPC9nPgo8IS0tIGRpc3BhdGNoZXImIzQ1OyZndDtncHVfcXVldWUgLS0+CjxnIGlkPSJlZGdlMyIgY2xhc3M9ImVkZ2UiPgo8dGl0bGU+ZGlzcGF0Y2hlciYjNDU7Jmd0O2dwdV9xdWV1ZTwvdGl0bGU+CjxwYXRoIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzQ0NDQ0NCIgZD0iTTY0Ni4zNiwtMzgyLjg5QzYzOS40MSwtMzc4LjE0IDYzMy4xMywtMzcyLjI0IDYyOSwtMzY1IDYyMy44MiwtMzU1LjkxIDYyNy45MywtMzQ4LjAzIDYzNS44OSwtMzQxLjUxIi8+Cjxwb2x5Z29uIGZpbGw9IiM0NDQ0NDQiIHN0cm9rZT0iIzQ0NDQ0NCIgcG9pbnRzPSI2MzcuNjYsLTM0My4yNiA2NDEuOTUsLTMzNy4yMSA2MzQuODMsLTMzOS4yNiA2MzcuNjYsLTM0My4yNiIvPgo8dGV4dCB0ZXh0LWFuY2hvcj0ibWlkZGxlIiB4PSI2NzQuNSIgeT0iLTM1Ny44IiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYiIGZvbnQtc2l6ZT0iOS4wMCI+Y29tcHV0ZV90eXBlPWdwdTwvdGV4dD4KPC9nPgo8IS0tIGNwdV9xdWV1ZSAtLT4KPGcgaWQ9Im5vZGU1IiBjbGFzcz0ibm9kZSI+Cjx0aXRsZT5jcHVfcXVldWU8L3RpdGxlPgo8cGF0aCBmaWxsPSIjZWNmOWVmIiBzdHJva2U9IiMyMjIyMjIiIGQ9Ik04NTQsLTMzN0M4NTQsLTMzNyA3NzYsLTMzNyA3NzYsLTMzNyA3NzAsLTMzNyA3NjQsLTMzMSA3NjQsLTMyNSA3NjQsLTMyNSA3NjQsLTMxMyA3NjQsLTMxMyA3NjQsLTMwNyA3NzAsLTMwMSA3NzYsLTMwMSA3NzYsLTMwMSA4NTQsLTMwMSA4NTQsLTMwMSA4NjAsLTMwMSA4NjYsLTMwNyA4NjYsLTMxMyA4NjYsLTMxMyA4NjYsLTMyNSA4NjYsLTMyNSA4NjYsLTMzMSA4NjAsLTMzNyA4NTQsLTMzNyIvPgo8dGV4dCB0ZXh0LWFuY2hvcj0ibWlkZGxlIiB4PSI4MTUiIHk9Ii0zMTYuNSIgZm9udC1mYW1pbHk9IkhlbHZldGljYSxzYW5zLVNlcmlmIiBmb250LXNpemU9IjEwLjAwIj5CYXRjaCBDUFUgUXVldWU8L3RleHQ+CjwvZz4KPCEtLSBkaXNwYXRjaGVyJiM0NTsmZ3Q7Y3B1X3F1ZXVlIC0tPgo8ZyBpZD0iZWRnZTQiIGNsYXNzPSJlZGdlIj4KPHRpdGxlPmRpc3BhdGNoZXImIzQ1OyZndDtjcHVfcXVldWU8L3RpdGxlPgo8cGF0aCBmaWxsPSJub25lIiBzdHJva2U9IiM0NDQ0NDQiIGQ9Ik03MTUuMjQsLTM4Mi44NEM3MzQuNzUsLTM3MC41NSA3NjEuMTEsLTM1My45NSA3ODEuODQsLTM0MC44OSIvPgo8cG9seWdvbiBmaWxsPSIjNDQ0NDQ0IiBzdHJva2U9IiM0NDQ0NDQiIHBvaW50cz0iNzgzLjE4LC0zNDIuOTQgNzg3LjgsLTMzNy4xMyA3ODAuNTcsLTMzOC43OSA3ODMuMTgsLTM0Mi45NCIvPgo8dGV4dCB0ZXh0LWFuY2hvcj0ibWlkZGxlIiB4PSI4MDQuNSIgeT0iLTM1Ny44IiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYiIGZvbnQtc2l6ZT0iOS4wMCI+Y29tcHV0ZV90eXBlPWNwdTwvdGV4dD4KPC9nPgo8IS0tIGdwdV9jZSAtLT4KPGcgaWQ9Im5vZGU2IiBjbGFzcz0ibm9kZSI+Cjx0aXRsZT5ncHVfY2U8L3RpdGxlPgo8cGF0aCBmaWxsPSIjZWNmOWVmIiBzdHJva2U9IiMyMjIyMjIiIGQ9Ik02NzksLTI2NEM2NzksLTI2NCA1NzcsLTI2NCA1NzcsLTI2NCA1NzEsLTI2NCA1NjUsLTI1OCA1NjUsLTI1MiA1NjUsLTI1MiA1NjUsLTI0MCA1NjUsLTI0MCA1NjUsLTIzNCA1NzEsLTIyOCA1NzcsLTIyOCA1NzcsLTIyOCA2NzksLTIyOCA2NzksLTIyOCA2ODUsLTIyOCA2OTEsLTIzNCA2OTEsLTI0MCA2OTEsLTI0MCA2OTEsLTI1MiA2OTEsLTI1MiA2OTEsLTI1OCA2ODUsLTI2NCA2NzksLTI2NCIvPgo8dGV4dCB0ZXh0LWFuY2hvcj0ibWlkZGxlIiB4PSI2MjgiIHk9Ii0yNDkiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1zaXplPSIxMC4wMCI+QVdTIEJhdGNoIEdQVTwvdGV4dD4KPHRleHQgdGV4dC1hbmNob3I9Im1pZGRsZSIgeD0iNjI4IiB5PSItMjM4IiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYiIGZvbnQtc2l6ZT0iMTAuMDAiPkNvbXB1dGUgRW52aXJvbm1lbnQ8L3RleHQ+CjwvZz4KPCEtLSBncHVfcXVldWUmIzQ1OyZndDtncHVfY2UgLS0+CjxnIGlkPSJlZGdlNSIgY2xhc3M9ImVkZ2UiPgo8dGl0bGU+Z3B1X3F1ZXVlJiM0NTsmZ3Q7Z3B1X2NlPC90aXRsZT4KPHBhdGggZmlsbD0ibm9uZSIgc3Ryb2tlPSIjNDQ0NDQ0IiBkPSJNNjcyLjcyLC0zMDAuODFDNjY0Ljg5LC0yOTEuMzkgNjU1LjEsLTI3OS42MSA2NDYuNjcsLTI2OS40NyIvPgo8cG9seWdvbiBmaWxsPSIjNDQ0NDQ0IiBzdHJva2U9IiM0NDQ0NDQiIHBvaW50cz0iNjQ4LjUxLC0yNjcuODUgNjQyLjE1LC0yNjQuMDMgNjQ0Ljc0LC0yNzAuOTggNjQ4LjUxLC0yNjcuODUiLz4KPC9nPgo8IS0tIGNwdV9jZSAtLT4KPGcgaWQ9Im5vZGU3IiBjbGFzcz0ibm9kZSI+Cjx0aXRsZT5jcHVfY2U8L3RpdGxlPgo8cGF0aCBmaWxsPSIjZWNmOWVmIiBzdHJva2U9IiMyMjIyMjIiIGQ9Ik04MzAsLTI2NEM4MzAsLTI2NCA3MjgsLTI2NCA3MjgsLTI2NCA3MjIsLTI2NCA3MTYsLTI1OCA3MTYsLTI1MiA3MTYsLTI1MiA3MTYsLTI0MCA3MTYsLTI0MCA3MTYsLTIzNCA3MjIsLTIyOCA3MjgsLTIyOCA3MjgsLTIyOCA4MzAsLTIyOCA4MzAsLTIyOCA4MzYsLTIyOCA4NDIsLTIzNCA4NDIsLTI0MCA4NDIsLTI0MCA4NDIsLTI1MiA4NDIsLTI1MiA4NDIsLTI1OCA4MzYsLTI2NCA4MzAsLTI2NCIvPgo8dGV4dCB0ZXh0LWFuY2hvcj0ibWlkZGxlIiB4PSI3NzkiIHk9Ii0yNDkiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1zaXplPSIxMC4wMCI+QVdTIEJhdGNoIENQVTwvdGV4dD4KPHRleHQgdGV4dC1hbmNob3I9Im1pZGRsZSIgeD0iNzc5IiB5PSItMjM4IiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYiIGZvbnQtc2l6ZT0iMTAuMDAiPkNvbXB1dGUgRW52aXJvbm1lbnQ8L3RleHQ+CjwvZz4KPCEtLSBjcHVfcXVldWUmIzQ1OyZndDtjcHVfY2UgLS0+CjxnIGlkPSJlZGdlNiIgY2xhc3M9ImVkZ2UiPgo8dGl0bGU+Y3B1X3F1ZXVlJiM0NTsmZ3Q7Y3B1X2NlPC90aXRsZT4KPHBhdGggZmlsbD0ibm9uZSIgc3Ryb2tlPSIjNDQ0NDQ0IiBkPSJNODA2LjI5LC0zMDAuODFDODAxLjY0LC0yOTEuNjYgNzk1Ljg2LC0yODAuMjYgNzkwLjgyLC0yNzAuMzIiLz4KPHBvbHlnb24gZmlsbD0iIzQ0NDQ0NCIgc3Ryb2tlPSIjNDQ0NDQ0IiBwb2ludHM9Ijc5Mi45OSwtMjY5LjE2IDc4Ny42MywtMjY0LjAzIDc4OC42MSwtMjcxLjM4IDc5Mi45OSwtMjY5LjE2Ii8+CjwvZz4KPCEtLSBncHVfam9iIC0tPgo8ZyBpZD0ibm9kZTgiIGNsYXNzPSJub2RlIj4KPHRpdGxlPmdwdV9qb2I8L3RpdGxlPgo8cGF0aCBmaWxsPSIjZWNmOWVmIiBzdHJva2U9IiMyMjIyMjIiIGQ9Ik0zNTkuNSwtMTkxQzM1OS41LC0xOTEgMjY2LjUsLTE5MSAyNjYuNSwtMTkxIDI2MC41LC0xOTEgMjU0LjUsLTE4NSAyNTQuNSwtMTc5IDI1NC41LC0xNzkgMjU0LjUsLTE2NyAyNTQuNSwtMTY3IDI1NC41LC0xNjEgMjYwLjUsLTE1NSAyNjYuNSwtMTU1IDI2Ni41LC0xNTUgMzU5LjUsLTE1NSAzNTkuNSwtMTU1IDM2NS41LC0xNTUgMzcxLjUsLTE2MSAzNzEuNSwtMTY3IDM3MS41LC0xNjcgMzcxLjUsLTE3OSAzNzEuNSwtMTc5IDM3MS41LC0xODUgMzY1LjUsLTE5MSAzNTkuNSwtMTkxIi8+Cjx0ZXh0IHRleHQtYW5jaG9yPSJtaWRkbGUiIHg9IjMxMyIgeT0iLTE3NiIgZm9udC1mYW1pbHk9IkhlbHZldGljYSxzYW5zLVNlcmlmIiBmb250LXNpemU9IjEwLjAwIj5Db250YWluZXIgRW50cnlwb2ludDwvdGV4dD4KPHRleHQgdGV4dC1hbmNob3I9Im1pZGRsZSIgeD0iMzEzIiB5PSItMTY1IiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYiIGZvbnQtc2l6ZT0iMTAuMDAiPisgSm9iIFNjcmlwdDwvdGV4dD4KPC9nPgo8IS0tIGdwdV9jZSYjNDU7Jmd0O2dwdV9qb2IgLS0+CjxnIGlkPSJlZGdlNyIgY2xhc3M9ImVkZ2UiPgo8dGl0bGU+Z3B1X2NlJiM0NTsmZ3Q7Z3B1X2pvYjwvdGl0bGU+CjxwYXRoIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzQ0NDQ0NCIgZD0iTTU2NC42MywtMjMwLjcyQzUxMC42MiwtMjE4LjU0IDQzMy4yOCwtMjAxLjExIDM3OC42MywtMTg4Ljc5Ii8+Cjxwb2x5Z29uIGZpbGw9IiM0NDQ0NDQiIHN0cm9rZT0iIzQ0NDQ0NCIgcG9pbnRzPSIzNzkuMTEsLTE4Ni4zOSAzNzEuNzQsLTE4Ny4yNCAzNzguMDMsLTE5MS4xNyAzNzkuMTEsLTE4Ni4zOSIvPgo8L2c+CjwhLS0gZWIgLS0+CjxnIGlkPSJub2RlMTUiIGNsYXNzPSJub2RlIj4KPHRpdGxlPmViPC90aXRsZT4KPHBhdGggZmlsbD0iI2U5ZjNmZiIgc3Ryb2tlPSIjMjIyMjIyIiBkPSJNNzYxLC0xOTFDNzYxLC0xOTEgNjY3LC0xOTEgNjY3LC0xOTEgNjYxLC0xOTEgNjU1LC0xODUgNjU1LC0xNzkgNjU1LC0xNzkgNjU1LC0xNjcgNjU1LC0xNjcgNjU1LC0xNjEgNjYxLC0xNTUgNjY3LC0xNTUgNjY3LC0xNTUgNzYxLC0xNTUgNzYxLC0xNTUgNzY3LC0xNTUgNzczLC0xNjEgNzczLC0xNjcgNzczLC0xNjcgNzczLC0xNzkgNzczLC0xNzkgNzczLC0xODUgNzY3LC0xOTEgNzYxLC0xOTEiLz4KPHRleHQgdGV4dC1hbmNob3I9Im1pZGRsZSIgeD0iNzE0IiB5PSItMTc2IiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYiIGZvbnQtc2l6ZT0iMTAuMDAiPkV2ZW50QnJpZGdlPC90ZXh0Pgo8dGV4dCB0ZXh0LWFuY2hvcj0ibWlkZGxlIiB4PSI3MTQiIHk9Ii0xNjUiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1zaXplPSIxMC4wMCI+QmF0Y2ggU3RhdGUgQ2hhbmdlczwvdGV4dD4KPC9nPgo8IS0tIGdwdV9jZSYjNDU7Jmd0O2ViIC0tPgo8ZyBpZD0iZWRnZTE3IiBjbGFzcz0iZWRnZSI+Cjx0aXRsZT5ncHVfY2UmIzQ1OyZndDtlYjwvdGl0bGU+CjxwYXRoIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzQ0NDQ0NCIgZD0iTTY0OC44MiwtMjI3LjgxQzY2MC41NCwtMjE4LjEzIDY3NS4zLC0yMDUuOTUgNjg3LjgxLC0xOTUuNjMiLz4KPHBvbHlnb24gZmlsbD0iIzQ0NDQ0NCIgc3Ryb2tlPSIjNDQ0NDQ0IiBwb2ludHM9IjY4OS41NCwtMTk3LjM3IDY5My4zNywtMTkxLjAzIDY4Ni40MiwtMTkzLjYgNjg5LjU0LC0xOTcuMzciLz4KPC9nPgo8IS0tIGNwdV9qb2IgLS0+CjxnIGlkPSJub2RlOSIgY2xhc3M9Im5vZGUiPgo8dGl0bGU+Y3B1X2pvYjwvdGl0bGU+CjxwYXRoIGZpbGw9IiNlY2Y5ZWYiIHN0cm9rZT0iIzIyMjIyMiIgZD0iTTU2Ni41LC0xOTFDNTY2LjUsLTE5MSA0NzMuNSwtMTkxIDQ3My41LC0xOTEgNDY3LjUsLTE5MSA0NjEuNSwtMTg1IDQ2MS41LC0xNzkgNDYxLjUsLTE3OSA0NjEuNSwtMTY3IDQ2MS41LC0xNjcgNDYxLjUsLTE2MSA0NjcuNSwtMTU1IDQ3My41LC0xNTUgNDczLjUsLTE1NSA1NjYuNSwtMTU1IDU2Ni41LC0xNTUgNTcyLjUsLTE1NSA1NzguNSwtMTYxIDU3OC41LC0xNjcgNTc4LjUsLTE2NyA1NzguNSwtMTc5IDU3OC41LC0xNzkgNTc4LjUsLTE4NSA1NzIuNSwtMTkxIDU2Ni41LC0xOTEiLz4KPHRleHQgdGV4dC1hbmNob3I9Im1pZGRsZSIgeD0iNTIwIiB5PSItMTc2IiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYiIGZvbnQtc2l6ZT0iMTAuMDAiPkNvbnRhaW5lciBFbnRyeXBvaW50PC90ZXh0Pgo8dGV4dCB0ZXh0LWFuY2hvcj0ibWlkZGxlIiB4PSI1MjAiIHk9Ii0xNjUiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1zaXplPSIxMC4wMCI+KyBKb2IgU2NyaXB0PC90ZXh0Pgo8L2c+CjwhLS0gY3B1X2NlJiM0NTsmZ3Q7Y3B1X2pvYiAtLT4KPGcgaWQ9ImVkZ2U4IiBjbGFzcz0iZWRnZSI+Cjx0aXRsZT5jcHVfY2UmIzQ1OyZndDtjcHVfam9iPC90aXRsZT4KPHBhdGggZmlsbD0ibm9uZSIgc3Ryb2tlPSIjNDQ0NDQ0IiBkPSJNNzE2Ljk2LC0yMjcuOTlDNjc3LjM1LC0yMTcuMTMgNjI2LjA2LC0yMDMuMDcgNTg1Ljc0LC0xOTIuMDIiLz4KPHBvbHlnb24gZmlsbD0iIzQ0NDQ0NCIgc3Ryb2tlPSIjNDQ0NDQ0IiBwb2ludHM9IjU4Ni4yNiwtMTg5LjYyIDU3OC44NiwtMTkwLjE0IDU4NC45NywtMTk0LjM1IDU4Ni4yNiwtMTg5LjYyIi8+CjwvZz4KPCEtLSBjcHVfY2UmIzQ1OyZndDtlYiAtLT4KPGcgaWQ9ImVkZ2UxOCIgY2xhc3M9ImVkZ2UiPgo8dGl0bGU+Y3B1X2NlJiM0NTsmZ3Q7ZWI8L3RpdGxlPgo8cGF0aCBmaWxsPSJub25lIiBzdHJva2U9IiM0NDQ0NDQiIGQ9Ik03NjMuMjcsLTIyNy44MUM3NTQuNjQsLTIxOC4zOSA3NDMuODUsLTIwNi42MSA3MzQuNTcsLTE5Ni40NyIvPgo8cG9seWdvbiBmaWxsPSIjNDQ0NDQ0IiBzdHJva2U9IiM0NDQ0NDQiIHBvaW50cz0iNzM2LjEyLC0xOTQuNTQgNzI5LjU5LC0xOTEuMDMgNzMyLjUxLC0xOTcuODUgNzM2LjEyLC0xOTQuNTQiLz4KPC9nPgo8IS0tIHMzX2lucHV0IC0tPgo8ZyBpZD0ibm9kZTEwIiBjbGFzcz0ibm9kZSI+Cjx0aXRsZT5zM19pbnB1dDwvdGl0bGU+CjxwYXRoIGZpbGw9IiNmZmY0ZGMiIHN0cm9rZT0iIzIyMjIyMiIgZD0iTTI0NywtMTA5QzI0NywtMTA5IDE3OSwtMTA5IDE3OSwtMTA5IDE3MywtMTA5IDE2NywtMTAzIDE2NywtOTcgMTY3LC05NyAxNjcsLTg1IDE2NywtODUgMTY3LC03OSAxNzMsLTczIDE3OSwtNzMgMTc5LC03MyAyNDcsLTczIDI0NywtNzMgMjUzLC03MyAyNTksLTc5IDI1OSwtODUgMjU5LC04NSAyNTksLTk3IDI1OSwtOTcgMjU5LC0xMDMgMjUzLC0xMDkgMjQ3LC0xMDkiLz4KPHRleHQgdGV4dC1hbmNob3I9Im1pZGRsZSIgeD0iMjEzIiB5PSItODguNSIgZm9udC1mYW1pbHk9IkhlbHZldGljYSxzYW5zLVNlcmlmIiBmb250LXNpemU9IjEwLjAwIj5TMyBJbnB1dCBCdWNrZXQ8L3RleHQ+CjwvZz4KPCEtLSBncHVfam9iJiM0NTsmZ3Q7czNfaW5wdXQgLS0+CjxnIGlkPSJlZGdlOSIgY2xhc3M9ImVkZ2UiPgo8dGl0bGU+Z3B1X2pvYiYjNDU7Jmd0O3MzX2lucHV0PC90aXRsZT4KPHBhdGggZmlsbD0ibm9uZSIgc3Ryb2tlPSIjNDQ0NDQ0IiBkPSJNMjkxLjMsLTE1NC42NEMyNzYuMjYsLTE0Mi42MSAyNTYuMTIsLTEyNi41IDI0MC4wNCwtMTEzLjYzIi8+Cjxwb2x5Z29uIGZpbGw9IiM0NDQ0NDQiIHN0cm9rZT0iIzQ0NDQ0NCIgcG9pbnRzPSIyNDEuMzEsLTExMS41MSAyMzQuMzEsLTEwOS4wNSAyMzguMjUsLTExNS4zNCAyNDEuMzEsLTExMS41MSIvPgo8L2c+CjwhLS0gczNfb3V0cHV0IC0tPgo8ZyBpZD0ibm9kZTExIiBjbGFzcz0ibm9kZSI+Cjx0aXRsZT5zM19vdXRwdXQ8L3RpdGxlPgo8cGF0aCBmaWxsPSIjZmZmNGRjIiBzdHJva2U9IiMyMjIyMjIiIGQ9Ik00NjUuNSwtMzZDNDY1LjUsLTM2IDM4OC41LC0zNiAzODguNSwtMzYgMzgyLjUsLTM2IDM3Ni41LC0zMCAzNzYuNSwtMjQgMzc2LjUsLTI0IDM3Ni41LC0xMiAzNzYuNSwtMTIgMzc2LjUsLTYgMzgyLjUsMCAzODguNSwwIDM4OC41LDAgNDY1LjUsMCA0NjUuNSwwIDQ3MS41LDAgNDc3LjUsLTYgNDc3LjUsLTEyIDQ3Ny41LC0xMiA0NzcuNSwtMjQgNDc3LjUsLTI0IDQ3Ny41LC0zMCA0NzEuNSwtMzYgNDY1LjUsLTM2Ii8+Cjx0ZXh0IHRleHQtYW5jaG9yPSJtaWRkbGUiIHg9IjQyNyIgeT0iLTE1LjUiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1zaXplPSIxMC4wMCI+UzMgT3V0cHV0IEJ1Y2tldDwvdGV4dD4KPC9nPgo8IS0tIGdwdV9qb2ImIzQ1OyZndDtzM19vdXRwdXQgLS0+CjxnIGlkPSJlZGdlMTEiIGNsYXNzPSJlZGdlIj4KPHRpdGxlPmdwdV9qb2ImIzQ1OyZndDtzM19vdXRwdXQ8L3RpdGxlPgo8cGF0aCBmaWxsPSJub25lIiBzdHJva2U9IiM0NDQ0NDQiIGQ9Ik0zMTAuMTMsLTE1NC45NkMzMDcuNTQsLTEzMy44MiAzMDYuMzEsLTk3LjU5IDMyMywtNzMgMzM0LjE5LC01Ni41MSAzNTIuMDUsLTQ0LjczIDM2OS44MywtMzYuNDciLz4KPHBvbHlnb24gZmlsbD0iIzQ0NDQ0NCIgc3Ryb2tlPSIjNDQ0NDQ0IiBwb2ludHM9IjM3MC45NSwtMzguNjYgMzc2LjM3LC0zMy41OSAzNjguOTcsLTM0LjE3IDM3MC45NSwtMzguNjYiLz4KPC9nPgo8IS0tIHMzX21vZGVscyAtLT4KPGcgaWQ9Im5vZGUxMiIgY2xhc3M9Im5vZGUiPgo8dGl0bGU+czNfbW9kZWxzPC90aXRsZT4KPHBhdGggZmlsbD0iI2ZmZjRkYyIgc3Ryb2tlPSIjMjIyMjIyIiBkPSJNMTI5LjUsLTEwOUMxMjkuNSwtMTA5IDUwLjUsLTEwOSA1MC41LC0xMDkgNDQuNSwtMTA5IDM4LjUsLTEwMyAzOC41LC05NyAzOC41LC05NyAzOC41LC04NSAzOC41LC04NSAzOC41LC03OSA0NC41LC03MyA1MC41LC03MyA1MC41LC03MyAxMjkuNSwtNzMgMTI5LjUsLTczIDEzNS41LC03MyAxNDEuNSwtNzkgMTQxLjUsLTg1IDE0MS41LC04NSAxNDEuNSwtOTcgMTQxLjUsLTk3IDE0MS41LC0xMDMgMTM1LjUsLTEwOSAxMjkuNSwtMTA5Ii8+Cjx0ZXh0IHRleHQtYW5jaG9yPSJtaWRkbGUiIHg9IjkwIiB5PSItODguNSIgZm9udC1mYW1pbHk9IkhlbHZldGljYSxzYW5zLVNlcmlmIiBmb250LXNpemU9IjEwLjAwIj5TMyBNb2RlbHMgQnVja2V0PC90ZXh0Pgo8L2c+CjwhLS0gZ3B1X2pvYiYjNDU7Jmd0O3MzX21vZGVscyAtLT4KPGcgaWQ9ImVkZ2UxMyIgY2xhc3M9ImVkZ2UiPgo8dGl0bGU+Z3B1X2pvYiYjNDU7Jmd0O3MzX21vZGVsczwvdGl0bGU+CjxwYXRoIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzQ0NDQ0NCIgZD0iTTI2NS40MywtMTU0Ljk0QzIyOS45MSwtMTQyLjE5IDE4MS4zMSwtMTI0Ljc2IDE0NC4yNywtMTExLjQ3Ii8+Cjxwb2x5Z29uIGZpbGw9IiM0NDQ0NDQiIHN0cm9rZT0iIzQ0NDQ0NCIgcG9pbnRzPSIxNDQuOTEsLTEwOS4xIDEzNy40OSwtMTA5LjA0IDE0My4yNSwtMTEzLjcxIDE0NC45MSwtMTA5LjEiLz4KPC9nPgo8IS0tIHMzX3ZhdWx0IC0tPgo8ZyBpZD0ibm9kZTEzIiBjbGFzcz0ibm9kZSI+Cjx0aXRsZT5zM192YXVsdDwvdGl0bGU+CjxwYXRoIGZpbGw9IiNmZmY0ZGMiIHN0cm9rZT0iIzIyMjIyMiIgZD0iTTUzNiwtMTA5QzUzNiwtMTA5IDQ2OCwtMTA5IDQ2OCwtMTA5IDQ2MiwtMTA5IDQ1NiwtMTAzIDQ1NiwtOTcgNDU2LC05NyA0NTYsLTg1IDQ1NiwtODUgNDU2LC03OSA0NjIsLTczIDQ2OCwtNzMgNDY4LC03MyA1MzYsLTczIDUzNiwtNzMgNTQyLC03MyA1NDgsLTc5IDU0OCwtODUgNTQ4LC04NSA1NDgsLTk3IDU0OCwtOTcgNTQ4LC0xMDMgNTQyLC0xMDkgNTM2LC0xMDkiLz4KPHRleHQgdGV4dC1hbmNob3I9Im1pZGRsZSIgeD0iNTAyIiB5PSItODguNSIgZm9udC1mYW1pbHk9IkhlbHZldGljYSxzYW5zLVNlcmlmIiBmb250LXNpemU9IjEwLjAwIj5TMyBWYXVsdCBCdWNrZXQ8L3RleHQ+CjwvZz4KPCEtLSBncHVfam9iJiM0NTsmZ3Q7czNfdmF1bHQgLS0+CjxnIGlkPSJlZGdlMTQiIGNsYXNzPSJlZGdlIj4KPHRpdGxlPmdwdV9qb2ImIzQ1OyZndDtzM192YXVsdDwvdGl0bGU+CjxwYXRoIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzQ0NDQ0NCIgZD0iTTM2OC4zOCwtMTU0LjkzQzM4My45NywtMTQ5LjY2IDQwMC44MSwtMTQzLjUxIDQxNiwtMTM3IDQzMi42NywtMTI5Ljg2IDQ1MC41MiwtMTIwLjc3IDQ2NS42MiwtMTEyLjYzIi8+Cjxwb2x5Z29uIGZpbGw9IiM0NDQ0NDQiIHN0cm9rZT0iIzQ0NDQ0NCIgcG9pbnRzPSI0NjYuOTMsLTExNC43IDQ3MS45MSwtMTA5LjIxIDQ2NC41OSwtMTEwLjQgNDY2LjkzLC0xMTQuNyIvPgo8L2c+CjwhLS0gZWZzX2NhY2hlIC0tPgo8ZyBpZD0ibm9kZTE0IiBjbGFzcz0ibm9kZSI+Cjx0aXRsZT5lZnNfY2FjaGU8L3RpdGxlPgo8cGF0aCBmaWxsPSIjZmZmNGRjIiBzdHJva2U9IiMyMjIyMjIiIGQ9Ik00MTguNSwtMTA5QzQxOC41LC0xMDkgMzQ3LjUsLTEwOSAzNDcuNSwtMTA5IDM0MS41LC0xMDkgMzM1LjUsLTEwMyAzMzUuNSwtOTcgMzM1LjUsLTk3IDMzNS41LC04NSAzMzUuNSwtODUgMzM1LjUsLTc5IDM0MS41LC03MyAzNDcuNSwtNzMgMzQ3LjUsLTczIDQxOC41LC03MyA0MTguNSwtNzMgNDI0LjUsLTczIDQzMC41LC03OSA0MzAuNSwtODUgNDMwLjUsLTg1IDQzMC41LC05NyA0MzAuNSwtOTcgNDMwLjUsLTEwMyA0MjQuNSwtMTA5IDQxOC41LC0xMDkiLz4KPHRleHQgdGV4dC1hbmNob3I9Im1pZGRsZSIgeD0iMzgzIiB5PSItODguNSIgZm9udC1mYW1pbHk9IkhlbHZldGljYSxzYW5zLVNlcmlmIiBmb250LXNpemU9IjEwLjAwIj5FRlMgL29wdC9tb2RlbHM8L3RleHQ+CjwvZz4KPCEtLSBncHVfam9iJiM0NTsmZ3Q7ZWZzX2NhY2hlIC0tPgo8ZyBpZD0iZWRnZTE2IiBjbGFzcz0iZWRnZSI+Cjx0aXRsZT5ncHVfam9iJiM0NTsmZ3Q7ZWZzX2NhY2hlPC90aXRsZT4KPHBhdGggZmlsbD0ibm9uZSIgc3Ryb2tlPSIjNDQ0NDQ0IiBkPSJNMzMyLjgyLC0xNDkuMzVDMzQyLjM2LC0xMzguNDUgMzUzLjc2LC0xMjUuNDEgMzYzLjI5LC0xMTQuNTMiLz4KPHBvbHlnb24gZmlsbD0iIzQ0NDQ0NCIgc3Ryb2tlPSIjNDQ0NDQ0IiBwb2ludHM9IjMzMC45NSwtMTQ3Ljc2IDMyOC4xOSwtMTU0LjY0IDMzNC42NCwtMTUwLjk5IDMzMC45NSwtMTQ3Ljc2Ii8+Cjxwb2x5Z29uIGZpbGw9IiM0NDQ0NDQiIHN0cm9rZT0iIzQ0NDQ0NCIgcG9pbnRzPSIzNjUuMzIsLTExNS45MyAzNjguMDgsLTEwOS4wNSAzNjEuNjMsLTExMi43IDM2NS4zMiwtMTE1LjkzIi8+Cjx0ZXh0IHRleHQtYW5jaG9yPSJtaWRkbGUiIHg9IjM4MSIgeT0iLTEyOS44IiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYiIGZvbnQtc2l6ZT0iOS4wMCI+bW9kZWwgY2FjaGU8L3RleHQ+CjwvZz4KPCEtLSBjcHVfam9iJiM0NTsmZ3Q7czNfaW5wdXQgLS0+CjxnIGlkPSJlZGdlMTAiIGNsYXNzPSJlZGdlIj4KPHRpdGxlPmNwdV9qb2ImIzQ1OyZndDtzM19pbnB1dDwvdGl0bGU+CjxwYXRoIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzQ0NDQ0NCIgZD0iTTQ2MS4yMiwtMTYyLjIzQzQyNy4wNSwtMTU2LjA2IDM4My4zLC0xNDcuMzEgMzQ1LC0xMzcgMzE4LjE4LC0xMjkuNzggMjg4Ljg5LC0xMTkuOTkgMjY0LjczLC0xMTEuMzkiLz4KPHBvbHlnb24gZmlsbD0iIzQ0NDQ0NCIgc3Ryb2tlPSIjNDQ0NDQ0IiBwb2ludHM9IjI2NS40OCwtMTA5LjA2IDI1OC4wNiwtMTA5IDI2My44MiwtMTEzLjY3IDI2NS40OCwtMTA5LjA2Ii8+CjwvZz4KPCEtLSBjcHVfam9iJiM0NTsmZ3Q7czNfb3V0cHV0IC0tPgo8ZyBpZD0iZWRnZTEyIiBjbGFzcz0iZWRnZSI+Cjx0aXRsZT5jcHVfam9iJiM0NTsmZ3Q7czNfb3V0cHV0PC90aXRsZT4KPHBhdGggZmlsbD0ibm9uZSIgc3Ryb2tlPSIjNDQ0NDQ0IiBkPSJNNTM2Ljc2LC0xNTQuODZDNTU0LjgxLC0xMzQuMjcgNTc4LjcsLTk5LjIxIDU2MSwtNzMgNTQzLjkyLC00Ny43IDUxMi44LC0zNC4yNiA0ODQuODUsLTI3LjExIi8+Cjxwb2x5Z29uIGZpbGw9IiM0NDQ0NDQiIHN0cm9rZT0iIzQ0NDQ0NCIgcG9pbnRzPSI0ODUuMiwtMjQuNjcgNDc3LjgyLC0yNS40MiA0ODQuMDYsLTI5LjQ0IDQ4NS4yLC0yNC42NyIvPgo8L2c+CjwhLS0gY3B1X2pvYiYjNDU7Jmd0O3MzX3ZhdWx0IC0tPgo8ZyBpZD0iZWRnZTE1IiBjbGFzcz0iZWRnZSI+Cjx0aXRsZT5jcHVfam9iJiM0NTsmZ3Q7czNfdmF1bHQ8L3RpdGxlPgo8cGF0aCBmaWxsPSJub25lIiBzdHJva2U9IiM0NDQ0NDQiIGQ9Ik01MTYuMDksLTE1NC42NEM1MTMuNTUsLTE0My4zNSA1MTAuMiwtMTI4LjQ2IDUwNy40MSwtMTE2LjA0Ii8+Cjxwb2x5Z29uIGZpbGw9IiM0NDQ0NDQiIHN0cm9rZT0iIzQ0NDQ0NCIgcG9pbnRzPSI1MDkuNzYsLTExNS4zNCA1MDUuODQsLTEwOS4wNSA1MDQuOTgsLTExNi40MiA1MDkuNzYsLTExNS4zNCIvPgo8L2c+CjwhLS0gbW9uaXRvciAtLT4KPGcgaWQ9Im5vZGUxNiIgY2xhc3M9Im5vZGUiPgo8dGl0bGU+bW9uaXRvcjwvdGl0bGU+CjxwYXRoIGZpbGw9IiNlOWYzZmYiIHN0cm9rZT0iIzIyMjIyMiIgZD0iTTcyOSwtMTA5QzcyOSwtMTA5IDY1NywtMTA5IDY1NywtMTA5IDY1MSwtMTA5IDY0NSwtMTAzIDY0NSwtOTcgNjQ1LC05NyA2NDUsLTg1IDY0NSwtODUgNjQ1LC03OSA2NTEsLTczIDY1NywtNzMgNjU3LC03MyA3MjksLTczIDcyOSwtNzMgNzM1LC03MyA3NDEsLTc5IDc0MSwtODUgNzQxLC04NSA3NDEsLTk3IDc0MSwtOTcgNzQxLC0xMDMgNzM1LC0xMDkgNzI5LC0xMDkiLz4KPHRleHQgdGV4dC1hbmNob3I9Im1pZGRsZSIgeD0iNjkzIiB5PSItODguNSIgZm9udC1mYW1pbHk9IkhlbHZldGljYSxzYW5zLVNlcmlmIiBmb250LXNpemU9IjEwLjAwIj5MYW1iZGEgTW9uaXRvcjwvdGV4dD4KPC9nPgo8IS0tIGViJiM0NTsmZ3Q7bW9uaXRvciAtLT4KPGcgaWQ9ImVkZ2UxOSIgY2xhc3M9ImVkZ2UiPgo8dGl0bGU+ZWImIzQ1OyZndDttb25pdG9yPC90aXRsZT4KPHBhdGggZmlsbD0ibm9uZSIgc3Ryb2tlPSIjNDQ0NDQ0IiBkPSJNNzA5LjQ0LC0xNTQuNjRDNzA2LjQ4LC0xNDMuMzUgNzAyLjU3LC0xMjguNDYgNjk5LjMxLC0xMTYuMDQiLz4KPHBvbHlnb24gZmlsbD0iIzQ0NDQ0NCIgc3Ryb2tlPSIjNDQ0NDQ0IiBwb2ludHM9IjcwMS42MiwtMTE1LjIgNjk3LjQ4LC0xMDkuMDUgNjk2Ljg4LC0xMTYuNDQgNzAxLjYyLC0xMTUuMiIvPgo8L2c+CjwhLS0gbW9uaXRvciYjNDU7Jmd0O3MzX291dHB1dCAtLT4KPGcgaWQ9ImVkZ2UyMSIgY2xhc3M9ImVkZ2UiPgo8dGl0bGU+bW9uaXRvciYjNDU7Jmd0O3MzX291dHB1dDwvdGl0bGU+CjxwYXRoIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzQ0NDQ0NCIgZD0iTTY1NS44OSwtNzIuOTVDNjQwLjQxLC02Ni4zMyA2MjIuMDgsLTU5LjEyIDYwNSwtNTQgNTY1LjU3LC00Mi4xOCA1MjAsLTMzLjMxIDQ4NC44NCwtMjcuNDUiLz4KPHBvbHlnb24gZmlsbD0iIzQ0NDQ0NCIgc3Ryb2tlPSIjNDQ0NDQ0IiBwb2ludHM9IjQ4NC44MSwtMjQuOTYgNDc3LjUxLC0yNi4yNSA0ODQuMDIsLTI5LjggNDg0LjgxLC0yNC45NiIvPgo8L2c+CjwhLS0gbm90aWZ5X3RvcGljIC0tPgo8ZyBpZD0ibm9kZTE3IiBjbGFzcz0ibm9kZSI+Cjx0aXRsZT5ub3RpZnlfdG9waWM8L3RpdGxlPgo8cGF0aCBmaWxsPSIjZTlmM2ZmIiBzdHJva2U9IiMyMjIyMjIiIGQ9Ik03NDIuNSwtMzZDNzQyLjUsLTM2IDY0My41LC0zNiA2NDMuNSwtMzYgNjM3LjUsLTM2IDYzMS41LC0zMCA2MzEuNSwtMjQgNjMxLjUsLTI0IDYzMS41LC0xMiA2MzEuNSwtMTIgNjMxLjUsLTYgNjM3LjUsMCA2NDMuNSwwIDY0My41LDAgNzQyLjUsMCA3NDIuNSwwIDc0OC41LDAgNzU0LjUsLTYgNzU0LjUsLTEyIDc1NC41LC0xMiA3NTQuNSwtMjQgNzU0LjUsLTI0IDc1NC41LC0zMCA3NDguNSwtMzYgNzQyLjUsLTM2Ii8+Cjx0ZXh0IHRleHQtYW5jaG9yPSJtaWRkbGUiIHg9IjY5MyIgeT0iLTE1LjUiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1zaXplPSIxMC4wMCI+U05TIE5vdGlmaWNhdGlvbiBUb3BpYzwvdGV4dD4KPC9nPgo8IS0tIG1vbml0b3ImIzQ1OyZndDtub3RpZnlfdG9waWMgLS0+CjxnIGlkPSJlZGdlMjAiIGNsYXNzPSJlZGdlIj4KPHRpdGxlPm1vbml0b3ImIzQ1OyZndDtub3RpZnlfdG9waWM8L3RpdGxlPgo8cGF0aCBmaWxsPSJub25lIiBzdHJva2U9IiM0NDQ0NDQiIGQ9Ik02OTMsLTcyLjgxQzY5MywtNjMuOTIgNjkzLC01Mi45MSA2OTMsLTQzLjE3Ii8+Cjxwb2x5Z29uIGZpbGw9IiM0NDQ0NDQiIHN0cm9rZT0iIzQ0NDQ0NCIgcG9pbnRzPSI2OTUuNDUsLTQzLjAzIDY5MywtMzYuMDMgNjkwLjU1LC00My4wMyA2OTUuNDUsLTQzLjAzIi8+CjwvZz4KPCEtLSB2cGMgLS0+CjxnIGlkPSJub2RlMTgiIGNsYXNzPSJub2RlIj4KPHRpdGxlPnZwYzwvdGl0bGU+CjxwYXRoIGZpbGw9IiNmZmVmZjYiIHN0cm9rZT0iIzIyMjIyMiIgZD0iTTU5MCwtNDkyQzU5MCwtNDkyIDQ1OCwtNDkyIDQ1OCwtNDkyIDQ1MiwtNDkyIDQ0NiwtNDg2IDQ0NiwtNDgwIDQ0NiwtNDgwIDQ0NiwtNDY4IDQ0NiwtNDY4IDQ0NiwtNDYyIDQ1MiwtNDU2IDQ1OCwtNDU2IDQ1OCwtNDU2IDU5MCwtNDU2IDU5MCwtNDU2IDU5NiwtNDU2IDYwMiwtNDYyIDYwMiwtNDY4IDYwMiwtNDY4IDYwMiwtNDgwIDYwMiwtNDgwIDYwMiwtNDg2IDU5NiwtNDkyIDU5MCwtNDkyIi8+Cjx0ZXh0IHRleHQtYW5jaG9yPSJtaWRkbGUiIHg9IjUyNCIgeT0iLTQ3MS41IiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYiIGZvbnQtc2l6ZT0iMTAuMDAiPlZQQyArIFB1YmxpYyBTdWJuZXRzICsgSUdXPC90ZXh0Pgo8L2c+CjwhLS0gdnBjJiM0NTsmZ3Q7Z3B1X2NlIC0tPgo8ZyBpZD0iZWRnZTIyIiBjbGFzcz0iZWRnZSI+Cjx0aXRsZT52cGMmIzQ1OyZndDtncHVfY2U8L3RpdGxlPgo8cGF0aCBmaWxsPSJub25lIiBzdHJva2U9IiM3YTdhN2EiIHN0cm9rZS1kYXNoYXJyYXk9IjUsMiIgZD0iTTUyNS4wNCwtNDU1Ljc5QzUyNy45NSwtNDIwLjE5IDUzOC43OCwtMzM3LjIgNTc4LC0yODIgNTgyLjkxLC0yNzUuMDkgNTg5LjYzLC0yNjkuMDggNTk2LjU2LC0yNjQuMDgiLz4KPC9nPgo8IS0tIHZwYyYjNDU7Jmd0O2NwdV9jZSAtLT4KPGcgaWQ9ImVkZ2UyMyIgY2xhc3M9ImVkZ2UiPgo8dGl0bGU+dnBjJiM0NTsmZ3Q7Y3B1X2NlPC90aXRsZT4KPHBhdGggZmlsbD0ibm9uZSIgc3Ryb2tlPSIjN2E3YTdhIiBzdHJva2UtZGFzaGFycmF5PSI1LDIiIGQ9Ik01MzIuOTIsLTQ1NS45OEM1NTQuNjMsLTQxNC43NyA2MDkuNDYsLTMxMi4yIDYyMywtMzAxIDYzNy42OSwtMjg4Ljg2IDY3OS4yMiwtMjc0Ljc1IDcxNS43NCwtMjY0LjAxIi8+CjwvZz4KPCEtLSBpYW0gLS0+CjxnIGlkPSJub2RlMTkiIGNsYXNzPSJub2RlIj4KPHRpdGxlPmlhbTwvdGl0bGU+CjxwYXRoIGZpbGw9IiNmZmVmZjYiIHN0cm9rZT0iIzIyMjIyMiIgZD0iTTEwMjIuNSwtNDkyQzEwMjIuNSwtNDkyIDkyOS41LC00OTIgOTI5LjUsLTQ5MiA5MjMuNSwtNDkyIDkxNy41LC00ODYgOTE3LjUsLTQ4MCA5MTcuNSwtNDgwIDkxNy41LC00NjggOTE3LjUsLTQ2OCA5MTcuNSwtNDYyIDkyMy41LC00NTYgOTI5LjUsLTQ1NiA5MjkuNSwtNDU2IDEwMjIuNSwtNDU2IDEwMjIuNSwtNDU2IDEwMjguNSwtNDU2IDEwMzQuNSwtNDYyIDEwMzQuNSwtNDY4IDEwMzQuNSwtNDY4IDEwMzQuNSwtNDgwIDEwMzQuNSwtNDgwIDEwMzQuNSwtNDg2IDEwMjguNSwtNDkyIDEwMjIuNSwtNDkyIi8+Cjx0ZXh0IHRleHQtYW5jaG9yPSJtaWRkbGUiIHg9Ijk3NiIgeT0iLTQ3MS41IiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYiIGZvbnQtc2l6ZT0iMTAuMDAiPklBTSBSb2xlcyArIFBvbGljaWVzPC90ZXh0Pgo8L2c+CjwhLS0gaWFtJiM0NTsmZ3Q7ZGlzcGF0Y2hlciAtLT4KPGcgaWQ9ImVkZ2UyNCIgY2xhc3M9ImVkZ2UiPgo8dGl0bGU+aWFtJiM0NTsmZ3Q7ZGlzcGF0Y2hlcjwvdGl0bGU+CjxwYXRoIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzdhN2E3YSIgc3Ryb2tlLWRhc2hhcnJheT0iNSwyIiBkPSJNOTMwLjQ1LC00NTUuOThDOTIyLjA1LC00NTMuMTIgOTEzLjMsLTQ1MC4zMyA5MDUsLTQ0OCA4NTAuODcsLTQzMi44IDc4Ny45LC00MTkuOTggNzQzLjc4LC00MTEuNzYiLz4KPC9nPgo8IS0tIGlhbSYjNDU7Jmd0O2dwdV9jZSAtLT4KPGcgaWQ9ImVkZ2UyNiIgY2xhc3M9ImVkZ2UiPgo8dGl0bGU+aWFtJiM0NTsmZ3Q7Z3B1X2NlPC90aXRsZT4KPHBhdGggZmlsbD0ibm9uZSIgc3Ryb2tlPSIjN2E3YTdhIiBzdHJva2UtZGFzaGFycmF5PSI1LDIiIGQ9Ik05NzEuNTIsLTQ1NS45Qzk2MS40LC00MjAuODYgOTMzLjMyLC0zNDAuODQgODc5LC0zMDEgODE0LjksLTI1My45OCA3ODEuNjgsLTI4MC45IDcwNCwtMjY0IDY5OS43NywtMjYzLjA4IDY5NS40MSwtMjYyLjEyIDY5MS4wMiwtMjYxLjE2Ii8+CjwvZz4KPCEtLSBpYW0mIzQ1OyZndDtjcHVfY2UgLS0+CjxnIGlkPSJlZGdlMjciIGNsYXNzPSJlZGdlIj4KPHRpdGxlPmlhbSYjNDU7Jmd0O2NwdV9jZTwvdGl0bGU+CjxwYXRoIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzdhN2E3YSIgc3Ryb2tlLWRhc2hhcnJheT0iNSwyIiBkPSJNOTgwLjczLC00NTUuODdDOTg5Ljc5LC00MTguOTQgMTAwNS4wMSwtMzMxLjIyIDk2MSwtMjgyIDk0NS4zNCwtMjY0LjQ5IDg4OC4zNSwtMjU1LjU3IDg0Mi4yNiwtMjUxLjEzIi8+CjwvZz4KPCEtLSBpYW0mIzQ1OyZndDttb25pdG9yIC0tPgo8ZyBpZD0iZWRnZTI1IiBjbGFzcz0iZWRnZSI+Cjx0aXRsZT5pYW0mIzQ1OyZndDttb25pdG9yPC90aXRsZT4KPHBhdGggZmlsbD0ibm9uZSIgc3Ryb2tlPSIjN2E3YTdhIiBzdHJva2UtZGFzaGFycmF5PSI1LDIiIGQ9Ik0xMDA3LjIxLC00NTUuODVDMTAyNS4wOSwtNDQzLjUzIDEwNDQsLTQyNS4wNCAxMDQ0LC00MDIgMTA0NCwtNDAyIDEwNDQsLTQwMiAxMDQ0LC0xNzIgMTA0NCwtMTEwLjU5IDgzNy4yLC05Ni4zMiA3NDEuMDcsLTkzIi8+CjwvZz4KPCEtLSBiYXRjaF9kZWZzIC0tPgo8ZyBpZD0ibm9kZTIwIiBjbGFzcz0ibm9kZSI+Cjx0aXRsZT5iYXRjaF9kZWZzPC90aXRsZT4KPHBhdGggZmlsbD0iI2ZmZWZmNiIgc3Ryb2tlPSIjMjIyMjIyIiBkPSJNODgwLC00OTJDODgwLC00OTIgNzg2LC00OTIgNzg2LC00OTIgNzgwLC00OTIgNzc0LC00ODYgNzc0LC00ODAgNzc0LC00ODAgNzc0LC00NjggNzc0LC00NjggNzc0LC00NjIgNzgwLC00NTYgNzg2LC00NTYgNzg2LC00NTYgODgwLC00NTYgODgwLC00NTYgODg2LC00NTYgODkyLC00NjIgODkyLC00NjggODkyLC00NjggODkyLC00ODAgODkyLC00ODAgODkyLC00ODYgODg2LC00OTIgODgwLC00OTIiLz4KPHRleHQgdGV4dC1hbmNob3I9Im1pZGRsZSIgeD0iODMzIiB5PSItNDcxLjUiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1zaXplPSIxMC4wMCI+QmF0Y2ggSm9iIERlZmluaXRpb25zPC90ZXh0Pgo8L2c+CjwhLS0gYmF0Y2hfZGVmcyYjNDU7Jmd0O2dwdV9xdWV1ZSAtLT4KPGcgaWQ9ImVkZ2UyOCIgY2xhc3M9ImVkZ2UiPgo8dGl0bGU+YmF0Y2hfZGVmcyYjNDU7Jmd0O2dwdV9xdWV1ZTwvdGl0bGU+CjxwYXRoIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzdhN2E3YSIgc3Ryb2tlLWRhc2hhcnJheT0iNSwyIiBkPSJNODQxLjE1LC00NTUuOTVDODUxLjg0LC00MzAuODQgODY2LjYyLC0zODMuNzUgODQzLC0zNTUgODE2LjgzLC0zMjMuMTQgNzkyLjE4LC0zNDYuMjIgNzUyLC0zMzcgNzQ3LjY4LC0zMzYuMDEgNzQzLjIxLC0zMzQuOTQgNzM4LjczLC0zMzMuODMiLz4KPC9nPgo8IS0tIGJhdGNoX2RlZnMmIzQ1OyZndDtjcHVfcXVldWUgLS0+CjxnIGlkPSJlZGdlMjkiIGNsYXNzPSJlZGdlIj4KPHRpdGxlPmJhdGNoX2RlZnMmIzQ1OyZndDtjcHVfcXVldWU8L3RpdGxlPgo8cGF0aCBmaWxsPSJub25lIiBzdHJva2U9IiM3YTdhN2EiIHN0cm9rZS1kYXNoYXJyYXk9IjUsMiIgZD0iTTg0Ny43MywtNDU1Ljc1Qzg2Ni4xMiwtNDMyIDg5MywtMzg4LjM2IDg3NCwtMzU1IDg2OS44NywtMzQ3Ljc2IDg2My41OSwtMzQxLjg2IDg1Ni42NCwtMzM3LjExIi8+CjwvZz4KPCEtLSBidWNrZXRzIC0tPgo8ZyBpZD0ibm9kZTIxIiBjbGFzcz0ibm9kZSI+Cjx0aXRsZT5idWNrZXRzPC90aXRsZT4KPHBhdGggZmlsbD0iI2ZmZWZmNiIgc3Ryb2tlPSIjMjIyMjIyIiBkPSJNNDA5LC00OTJDNDA5LC00OTIgMzYzLC00OTIgMzYzLC00OTIgMzU3LC00OTIgMzUxLC00ODYgMzUxLC00ODAgMzUxLC00ODAgMzUxLC00NjggMzUxLC00NjggMzUxLC00NjIgMzU3LC00NTYgMzYzLC00NTYgMzYzLC00NTYgNDA5LC00NTYgNDA5LC00NTYgNDE1LC00NTYgNDIxLC00NjIgNDIxLC00NjggNDIxLC00NjggNDIxLC00ODAgNDIxLC00ODAgNDIxLC00ODYgNDE1LC00OTIgNDA5LC00OTIiLz4KPHRleHQgdGV4dC1hbmNob3I9Im1pZGRsZSIgeD0iMzg2IiB5PSItNDcxLjUiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2Esc2Fucy1TZXJpZiIgZm9udC1zaXplPSIxMC4wMCI+UzMgQnVja2V0czwvdGV4dD4KPC9nPgo8IS0tIGJ1Y2tldHMmIzQ1OyZndDtzM19pbnB1dCAtLT4KPGcgaWQ9ImVkZ2UzMCIgY2xhc3M9ImVkZ2UiPgo8dGl0bGU+YnVja2V0cyYjNDU7Jmd0O3MzX2lucHV0PC90aXRsZT4KPHBhdGggZmlsbD0ibm9uZSIgc3Ryb2tlPSIjN2E3YTdhIiBzdHJva2UtZGFzaGFycmF5PSI1LDIiIGQ9Ik0zNTUuNSwtNDU1Ljg3QzM1MC4wOCwtNDUzLjA5IDM0NC40MywtNDUwLjM1IDMzOSwtNDQ4IDI4MC44MywtNDIyLjgxIDIwNCwtNDY1LjM5IDIwNCwtNDAyIDIwNCwtNDAyIDIwNCwtNDAyIDIwNCwtMTcyIDIwNCwtMTUwLjIgMjA3LjM0LC0xMjUuMjYgMjA5Ljk1LC0xMDkuMSIvPgo8L2c+CjwhLS0gYnVja2V0cyYjNDU7Jmd0O3MzX291dHB1dCAtLT4KPGcgaWQ9ImVkZ2UzMSIgY2xhc3M9ImVkZ2UiPgo8dGl0bGU+YnVja2V0cyYjNDU7Jmd0O3MzX291dHB1dDwvdGl0bGU+CjxwYXRoIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzdhN2E3YSIgc3Ryb2tlLWRhc2hhcnJheT0iNSwyIiBkPSJNMzU4LjQ4LC00NTUuOTJDMzUyLjI3LC00NTIuNzcgMzQ1LjU3LC00NDkuODkgMzM5LC00NDggMjY1LjkyLC00MjcuMDQgMCwtNDc4LjAyIDAsLTQwMiAwLC00MDIgMCwtNDAyIDAsLTkwIDAsLTUyLjI0IDI2My4yMywtMzAgMzc2LjM0LC0yMi4xOSIvPgo8L2c+CjwhLS0gYnVja2V0cyYjNDU7Jmd0O3MzX21vZGVscyAtLT4KPGcgaWQ9ImVkZ2UzMiIgY2xhc3M9ImVkZ2UiPgo8dGl0bGU+YnVja2V0cyYjNDU7Jmd0O3MzX21vZGVsczwvdGl0bGU+CjxwYXRoIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzdhN2E3YSIgc3Ryb2tlLWRhc2hhcnJheT0iNSwyIiBkPSJNMzU4LjIsLTQ1NS44N0MzNTIuMDcsLTQ1Mi43NyAzNDUuNDcsLTQ0OS45MiAzMzksLTQ0OCAyOTEuNzMsLTQzNC4wMSAxMjIsLTQ1MS4yOSAxMjIsLTQwMiAxMjIsLTQwMiAxMjIsLTQwMiAxMjIsLTE3MiAxMjIsLTE0OC45NiAxMTAuNDIsLTEyNC42OSAxMDEuMTksLTEwOS4wMSIvPgo8L2c+CjwhLS0gYnVja2V0cyYjNDU7Jmd0O3MzX3ZhdWx0IC0tPgo8ZyBpZD0iZWRnZTMzIiBjbGFzcz0iZWRnZSI+Cjx0aXRsZT5idWNrZXRzJiM0NTsmZ3Q7czNfdmF1bHQ8L3RpdGxlPgo8cGF0aCBmaWxsPSJub25lIiBzdHJva2U9IiM3YTdhN2EiIHN0cm9rZS1kYXNoYXJyYXk9IjUsMiIgZD0iTTQwMC44NCwtNDU1LjlDNDExLjEzLC00NDIuMjYgNDIzLC00MjIuMTUgNDIzLC00MDIgNDIzLC00MDIgNDIzLC00MDIgNDIzLC0yMDkgNDIzLC0xNjcuNTggNDU3LjQzLC0xMjkuODYgNDgwLjgzLC0xMDkiLz4KPC9nPgo8IS0tIGxhbWJkYXMgLS0+CjxnIGlkPSJub2RlMjIiIGNsYXNzPSJub2RlIj4KPHRpdGxlPmxhbWJkYXM8L3RpdGxlPgo8cGF0aCBmaWxsPSIjZmZlZmY2IiBzdHJva2U9IiMyMjIyMjIiIGQ9Ik03MzYuNSwtNDkyQzczNi41LC00OTIgNjM5LjUsLTQ5MiA2MzkuNSwtNDkyIDYzMy41LC00OTIgNjI3LjUsLTQ4NiA2MjcuNSwtNDgwIDYyNy41LC00ODAgNjI3LjUsLTQ2OCA2MjcuNSwtNDY4IDYyNy41LC00NjIgNjMzLjUsLTQ1NiA2MzkuNSwtNDU2IDYzOS41LC00NTYgNzM2LjUsLTQ1NiA3MzYuNSwtNDU2IDc0Mi41LC00NTYgNzQ4LjUsLTQ2MiA3NDguNSwtNDY4IDc0OC41LC00NjggNzQ4LjUsLTQ4MCA3NDguNSwtNDgwIDc0OC41LC00ODYgNzQyLjUsLTQ5MiA3MzYuNSwtNDkyIi8+Cjx0ZXh0IHRleHQtYW5jaG9yPSJtaWRkbGUiIHg9IjY4OCIgeT0iLTQ3MS41IiBmb250LWZhbWlseT0iSGVsdmV0aWNhLHNhbnMtU2VyaWYiIGZvbnQtc2l6ZT0iMTAuMDAiPkRpc3BhdGNoZXIgKyBNb25pdG9yPC90ZXh0Pgo8L2c+CjwhLS0gbGFtYmRhcyYjNDU7Jmd0O2Rpc3BhdGNoZXIgLS0+CjxnIGlkPSJlZGdlMzQiIGNsYXNzPSJlZGdlIj4KPHRpdGxlPmxhbWJkYXMmIzQ1OyZndDtkaXNwYXRjaGVyPC90aXRsZT4KPHBhdGggZmlsbD0ibm9uZSIgc3Ryb2tlPSIjN2E3YTdhIiBzdHJva2UtZGFzaGFycmF5PSI1LDIiIGQ9Ik02ODgsLTQ1NS44MUM2ODgsLTQ0NC42NSA2ODgsLTQzMC4xNiA2ODgsLTQxOS4wMyIvPgo8L2c+CjwhLS0gbGFtYmRhcyYjNDU7Jmd0O21vbml0b3IgLS0+CjxnIGlkPSJlZGdlMzUiIGNsYXNzPSJlZGdlIj4KPHRpdGxlPmxhbWJkYXMmIzQ1OyZndDttb25pdG9yPC90aXRsZT4KPHBhdGggZmlsbD0ibm9uZSIgc3Ryb2tlPSIjN2E3YTdhIiBzdHJva2UtZGFzaGFycmF5PSI1LDIiIGQ9Ik02NjAuNDcsLTQ1NS44N0M2NDYuODksLTQ0Ni4zMyA2MzEuMDksLTQzMy40OCA2MjAsLTQxOSA1NjUuMjgsLTM0Ny41OSA1MTcuOTcsLTMxMC44NiA1NTMsLTIyOCA1NTMuMSwtMjI3Ljc2IDYzNy42NCwtMTQ1LjcxIDY3NS4yOCwtMTA5LjE5Ii8+CjwvZz4KPC9nPgo8L3N2Zz4K" />
<p>Four S3 buckets:</p>
<pre><code>- input (scripts and payloads)
- output (results)
- models (cached HuggingFace weights)
- vault (cookies and API keys)</code></pre>
<p>An EFS volume is mounted at <code>/opt/models</code> across Batch instances so models downloaded on one job are cached for the next.</p>
<p>The VPC uses public subnets with an internet gateway rather than private subnets with a NAT gateway. Batch instances need to reach ECR to pull images and S3 for data. NAT gateways run ~$32/month even with no traffic. Public subnets with proper security groups accomplish the same thing at no standing cost.</p>
<h1 data-number="3" id="trigger-flow"><span class="header-section-number">3</span> Trigger Flow</h1>
<p>A job submission is a JSON message published to the SNS trigger topic:</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode json"><code class="sourceCode json"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="fu">{</span></span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a> <span class="dt">"trigger_type"</span><span class="fu">:</span> <span class="st">"batch_job"</span><span class="fu">,</span></span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a> <span class="dt">"data"</span><span class="fu">:</span> <span class="fu">{</span></span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a> <span class="dt">"script_key"</span><span class="fu">:</span> <span class="st">"jobs/transcribe_processor.py"</span><span class="fu">,</span></span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a> <span class="dt">"compute_type"</span><span class="fu">:</span> <span class="st">"gpu"</span><span class="fu">,</span></span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a> <span class="dt">"input_key"</span><span class="fu">:</span> <span class="st">"audio/interview.wav"</span><span class="fu">,</span></span>
<span id="cb2-7"><a href="#cb2-7" aria-hidden="true" tabindex="-1"></a> <span class="dt">"output_key"</span><span class="fu">:</span> <span class="st">"transcripts/interview.json"</span></span>
<span id="cb2-8"><a href="#cb2-8" aria-hidden="true" tabindex="-1"></a> <span class="fu">},</span></span>
<span id="cb2-9"><a href="#cb2-9" aria-hidden="true" tabindex="-1"></a> <span class="dt">"metadata"</span><span class="fu">:</span> <span class="fu">{</span></span>
<span id="cb2-10"><a href="#cb2-10" aria-hidden="true" tabindex="-1"></a> <span class="dt">"user"</span><span class="fu">:</span> <span class="st">"nana"</span><span class="fu">,</span></span>
<span id="cb2-11"><a href="#cb2-11" aria-hidden="true" tabindex="-1"></a> <span class="dt">"project"</span><span class="fu">:</span> <span class="st">"transcription"</span></span>
<span id="cb2-12"><a href="#cb2-12" aria-hidden="true" tabindex="-1"></a> <span class="fu">}</span></span>
<span id="cb2-13"><a href="#cb2-13" aria-hidden="true" tabindex="-1"></a><span class="fu">}</span></span></code></pre></div>
<p>The Lambda dispatcher validates the message, resolves resource defaults (vCPUs, memory, GPUs) from Terraform variables if the caller didn’t specify them, generates a unique job name, and submits it to the appropriate Batch queue. The full SNS payload is passed into the container via <code>SNS_MESSAGE</code> so the Python script sees everything the caller sent.</p>
<p>The entrypoint script inside the container downloads the <code>.py</code> file from S3 using <code>SCRIPT_KEY</code>, runs it, then uploads everything in <code>/workspace/output/</code> and <code>/workspace/logs/</code> back to S3.</p>
<p>A separate Lambda monitors Batch job state changes via EventBridge. When a job finishes, it sends an SNS notification and writes a summary JSON to S3.</p>
<h1 data-number="4" id="compute"><span class="header-section-number">4</span> Compute</h1>
<p>Two Batch compute environments:</p>
<ul>
<li><strong>GPU</strong>: g4dn.xlarge, g4dn.2xlarge, g5.xlarge. Spot by default. Allocation strategy is <code>SPOT_CAPACITY_OPTIMIZED</code>. Min vCPUs: 0, max: 256.</li>
<li><strong>CPU</strong>: m5.large, c6a.large, t3 variants. Same spot setup. Min: 0, max: 128.</li>
</ul>
<p>Default job resources are set in Terraform and can be overridden per job submission. GPU jobs default to 4 vCPU, 16GB RAM, 1 GPU. CPU jobs default to 2 vCPU, 4GB RAM.</p>
<h1 data-number="5" id="docker-images"><span class="header-section-number">5</span> Docker Images</h1>
<p>Two images built locally and pushed to ECR:</p>
<ul>
<li><strong>cpu-slim</strong>: Based on <code>python:3.11-slim</code>. Installs FFmpeg, Node.js, yt-dlp. Around 500MB. Builds in about 2 minutes.</li>
<li><strong>gpu-slim</strong>: Based on <code>nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04</code>. Installs Python 3.11, PyTorch 2.1, faster-whisper, pyannote.audio, transformers. Around 5GB. Takes 5-10 minutes to build.</li>
</ul>
<p>Getting CUDA, cuDNN, PyTorch, and pyannote versions to agree took several iterations. The current combination locks to CUDA 12.4.1 with cuDNN runtime and lets PyTorch detect GPU support at runtime.</p>
<h1 data-number="6" id="job-scripts"><span class="header-section-number">6</span> Job Scripts</h1>
<p>The pipeline does not require a fixed set of scripts. You can bring your own Python jobs as long as each one is uploaded to the input S3 bucket and referenced by <code>script_key</code> in the trigger payload.</p>
<p>The scripts below are examples from this project, not required components:</p>
<ul>
<li><strong>transcribe_processor.py</strong>: Runs faster-whisper for speech-to-text and pyannote for speaker diarization. Outputs JSON with segments, speaker labels, and timestamps. Downloads models from HuggingFace on first run and caches them on EFS.</li>
<li><strong>video_processor.py</strong>: Extracts audio from video using FFmpeg. Outputs 16kHz mono WAV normalized for Whisper input.</li>
<li><strong>download_processor.py</strong>: Downloads video from YouTube via yt-dlp. Reads cookies from the vault bucket for authenticated requests.</li>
<li><strong>scoring_processor.py</strong>: Scores transcript segments using an LLM (Bedrock, OpenAI, or Anthropic). Takes segments JSON as input, returns scored segments. Provider is pluggable.</li>
<li><strong>cleanup_processor.py</strong>: Deletes cached models from EFS. Useful when you want to free EFS storage between projects.</li>
</ul>
<p>In practice, the contract is simple: your script receives context via environment variables (<code>ML_INPUT_BUCKET</code>, <code>ML_OUTPUT_BUCKET</code>, <code>OUTPUT_PREFIX</code>, etc.), does whatever workload you need, and writes results to <code>/workspace/output/</code>. After the script exits, the container entrypoint syncs outputs (and logs) back to S3.</p>
<h1 data-number="7" id="admin-tools"><span class="header-section-number">7</span> Admin Tools</h1>
<p>A small FastAPI app (<code>admin/ui/app.py</code>) running locally at port 8000 gives a UI for building and submitting job payloads. It has presets for common job types, a JSON override editor, an S3 bucket selector, and publishes directly to SNS.</p>
<p>CLI scripts in <code>admin/scripts/</code> cover the same ground plus a few utilities: uploading all job scripts to S3, downloading models to the vault bucket before the first run, and exporting Firefox cookies to the Netscape format yt-dlp expects.</p>
<h1 data-number="8" id="state-and-backend"><span class="header-section-number">8</span> State and Backend</h1>
<p>This project stores Terraform state remotely in S3 with DynamoDB locking, so state is shared and protected from concurrent writes. One practical detail is that backend configuration cannot use regular Terraform variables, so these values are passed as CLI flags during <code>terraform init</code>:</p>
<div class="sourceCode" id="cb3"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="ex">terraform</span> init <span class="dt">\</span></span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a> <span class="at">-backend-config</span><span class="op">=</span><span class="st">"bucket=</span><span class="va">${PROJECT_NAME}</span><span class="st">-terraform-state"</span> <span class="dt">\</span></span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a> <span class="at">-backend-config</span><span class="op">=</span><span class="st">"dynamodb_table=</span><span class="va">${PROJECT_NAME}</span><span class="st">-terraform-lock"</span> <span class="dt">\</span></span>
<span id="cb3-4"><a href="#cb3-4" aria-hidden="true" tabindex="-1"></a> <span class="at">-backend-config</span><span class="op">=</span><span class="st">"region=</span><span class="va">$AWS_REGION</span><span class="st">"</span></span></code></pre></div>
<h1 data-number="9" id="gpu-quotas"><span class="header-section-number">9</span> GPU Quotas</h1>
<p>AWS Batch can’t launch GPU instances if the account’s EC2 vCPU quota for G/VT instances is at the default (which is often 0 in new accounts). This isn’t manageable through Terraform. You have to request an increase through AWS Service Quotas.</p>
<p>Quota codes that are consistent across all accounts:</p>
<ul>
<li><code>L-DB2E81BA</code>: Running On-Demand G and VT instances</li>
<li><code>L-3819A6DF</code>: All G and VT Spot Instances</li>
</ul>
<p>Jobs will sit in <code>RUNNABLE</code> status indefinitely without sufficient quota.</p>
<h1 data-number="10" id="code"><span class="header-section-number">10</span> Code</h1>
<p><a href="https://github.com/hiram-labs/terraform-aws-ml">hiram-labs/terraform-aws-ml</a></p>
Comments
No comments yet
Be the first to comment!