SizeMyGenAINet - FAQ

• The Generative AI Sizer is a tool designed to help you determine the appropriate configuration and server resources needed for different Generative AI models. By providing insights into parameters and model requirements, it assists in optimizing costs and performance for training and inference tasks.

Selecting a Model Configuration:

• Navigate to the "Generative AI Model" section.
• Choose a predefined model from the "Generative AI Model" dropdown. This will automatically populate the model parameters for you and enable the "Training Dataset Size" dropdown.
• If you prefer, you can also manually adjust or enter the model parameters by clicking on "(show details)" next to the "Generative AI Model" header.

• After selecting your model example and/or editing the model parameters, set the training dataset size in the "Training Dataset Size" field. This represents the number of words/word pieces (AKA tokens) in your training dataset.

• Decide if you are training the model or only performing inferences.
• If training, select the desired "Batch Size". The sizer will recommend a number of micro-batches accordingly.
• If performing inferences only, choose the "Inference Only" option.

• Go to the "Server Configuration" section.
• Choose a predefined GPU configuration from the dropdown.
• For more advanced settings, click on "(show details)" next to the "Server Configuration" header and you can configure your own GPU. If your GPU does not have Tensor cores set the TFLOPs fields to the same value.

• The sizer will calculate and display the resource requirements in the "Model Sizing" section.
• Details include the number of parameters, maximum GPU memory required when training or recommended GPU memory required when inferencing, and quantity of servers required depending on optimization techniques used and the time to complete the Training Dataset Size with each..
• Refer to the chart below the results for a visual representation.

Tool Tips:

• Hover over question mark icons "(?)" for more information or clarifications on specific fields and values.

• You can adjust various model parameters including model example and its configuration including layers, heads, input length, hidden layer size or dimension and vocabulary size. When training you can select a dataset size and batch size. You can also adjust GPU configuration including GPU memory size, total GPUs in a server, the TFLOPS and Sparse TFLOPS values for the GPU.

• As of now, the sizer provides real-time results but doesn't offer a save feature. We recommend noting down or screenshotting important configurations for future reference.

• In the "Generative AI Model" section, use the dropdown menu labeled "Generative AI Model" to choose a predefined model. This will auto-populate the relevant parameters for you.

• Absolutely! Just click on "(show details)" next to the "Generative AI Model" header to adjust or enter parameters manually.

• Clicking on "(show details)" expands the section, revealing additional parameters and advanced configurations associated with the selected model.

• Selecting a batch size pertains to training the model, determining how many samples are processed as a group before validating model performance. Choosing "Inference Only" indicates you're only running the model for predictions, not training.

• The sizer identifies the number of micro-batches based on the chosen batch size and model configuration to complete the processing of a batch of input tokens.

• This section provides information on resource requirements, such as the number of parameters, maximum GPU memory needed, and server demands under varied memory distribution and optimization techniques. Use this to gauge the feasibility and costs of running your chosen configuration.

• These parameters help you understand the computational and memory demands of the model. For example, GPU memory indicates how much graphics could be occupied during the model's training assuming no optimization. During inference GPU Memory Required assumes Tensor Parallelism is implemented reducing the duplication of tensor values across distributed GPUs. These values help define the number of GPUs needed.

• The chart visually represents the resource requirements, allowing you to quickly gauge and compare different optimization techniques and their impact on overall cost to compute.

• The sizer provides recommendations based on generalized math formulas and published performance results. Specific datasets, model tweaks, or external conditions might lead to variations. Always use the recommendations as a starting point and adjust based on your specific needs.