JetPack¶
JetPack includes Jetson Linux with bootloader, Linux kernel, Ubuntu desktop environment, and a complete set of libraries for GPU computing, multimedia, graphics, and computer vision acceleration. It also includes examples, documentation, and developer tools for hosts and developer suites, and supports higher-level SDKs such as DeepStream for streaming video analysis, Isaac for robot development, and Riva for conversational AI.
Install¶
sudo apt update
sudo apt install nvidia-jetpack
TensorRT¶
NVIDIA TensorRT is an ecosystem of APIs for high-performance deep learning inference. TensorRT includes an inference runtime and model optimizations that deliver low latency and high throughput for production applications. The TensorRT ecosystem includes TensorRT, TensorRT-LLM, TensorRT Model Optimizer, and TensorRT Cloud. TensorRT Advantage:
Speed Up Inference by 36X
Optimize Inference Performance
Accelerate Every Workload
Deploy, Run, and Scale With Triton
Generative AI¶
NVIDIA Jetson Orin gives you unparalleled AI compute, large unified memory, and comprehensive software stacks, delivering superior energy efficiency to drive the latest generative AI applications. It delivers fast inference for any gen AI models powered by the transformer architecture, leading the edge performance on MLPerf. The Jetson AI Lab is your gateway to explore tutorials on text generation, text + vision models, image generation, distillation techniques, and access resources to run these models on NVIDIA Jetson Orin. Join the generative AI revolution and start today.
Text Generative: LLM¶
Model： llama-2-7b-chat.Q4_K_M.gguf
How to try¶
No.1¶
AIBOX-Orin NX and PC within the same local area network:
On AIBOX-Orin NX to run command:
jetson-containers run $(autotag text-generation-webui)On PC to open your browser and access
http://<AIBOX-Orin NX IP_ADDRESS>:7860
No.2¶
On AIBOX-Orin NX to run command:
jetson-containers run $(autotag text-generation-webui)On AIBOX-Orin NX to open your browser and access
http://0.0.0.0:7860
Load Model¶
On web UI：
Click “Model”
Click “🔄”
Select “TheBloke_Llama-2-7b-Chat-GGUF”
Set
n-gpu-layersto128Click “Load”
Image Generative: Stable Diffusion¶
Model： v1-5-pruned-emaonly.safetensors
How to try¶
No.1¶
AIBOX-Orin NX and PC within the same local area network:
On AIBOX-Orin NX to run command:
jetson-containers run $(autotag stable-diffusion-webui)On PC to open your browser and access
http://<AIBOX-Orin NX IP_ADDRESS>:7860
No.2¶
On AIBOX-Orin NX to run command:
jetson-containers run $(autotag stable-diffusion-webui)On AIBOX-Orin NX to open your browser and access
http://0.0.0.0:7860
Load Model¶
On web UI
