You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I must say that this embedding too actually does what it says, I was able to embed 8 million tokens in just 3 hours on my Macbook pro m1 which Ollama embedding has taken over 2 days now that I just had to end it.
But what happened is when I used a chat model to polish reference to human like responses, it was unable to pick some stuffs out of the context.
So my question would be, does it have a limit in how much it embeds, that it left some stuffs out?
Hey! No, we don't leave anything. The default model has a 512 tokens length size, and we honor that. We'd encourage that you keep the chunk size to be smaller than 512 tokens!
Activity
NirantK commentedon Nov 16, 2023
Hey! No, we don't leave anything. The default model has a 512 tokens length size, and we honor that. We'd encourage that you keep the chunk size to be smaller than 512 tokens!