Added Support for Apple Silicon

[shashikanth-a]

• Unoptimized
• No gguf support yet.
• Build Triton and bitsandbytes from source
• cmake -DCOMPUTE_BACKEND=mps -S . for bitsandbytes building
• pip install unsloth-zoo==2024.11.4
• pip install xformers==0.0.25

[yukiarimo]

Is this working?

[shimmyshimmer]

Hi there thank you for this we will need a bit more time to review! :)

[mkemka]

Hi @shashikanth-a - thank you for this. Could you please provide information about the environment and package versions you used for development?

[yukiarimo]

Hey, does this works with newly released vision support?

[mkemka]

Currently I can run this if:

• Decorators mentioning "@torch.compile(fullgraph = False, dynamic = True, options = torch_compile_options)" are removed in llama and Gemma files.
• Fine tune llama-3-8b (3.2 1b and 3b throw an error due to rope for some reason.

[mkemka]

With the changes I can run this out of the box with the steps outlined above:

• Build Triton from source and pip install -e .
• Build bnb with cmake -DCOMPUTE_BACKEND=mps -S . and pip install -e .

On a M4 Pro getting around 100 t/s for llama3-8b. Can confirm it will also now work with llama-3.2-3b

[shimmyshimmer]

Thanks a lot - would anyone be so kind to benchmark this against MLX itself and share results?

Time it took, amount of VRAM, context length, if the losses match - ofcourse it's a lot so just time and checking to see if the losses match would be more than helpful. Thank you so much! :)