best option for chunking data
large body of text, multiple files, inconsistent format. llms seem to be hit or miss when it comes to chunking. is there a application that I don't know about that can make it happen? the text is academic medical articles with tons of content. I want to chunk it for embedding purposes submitted by /u/Immediate_Occasion69 [link] [comments]
Could not retrieve the full article text.
Read on Reddit r/LocalLLaMA →Reddit r/LocalLLaMA
https://www.reddit.com/r/LocalLLaMA/comments/1saf6dd/best_option_for_chunking_data/Sign in to highlight and annotate this article
Conversation starters
Daily AI Digest
Get the top 5 AI stories delivered to your inbox every morning.
More about
application
This bike rack pioneer is selling Bluetooth suction cups to stick bikes to your car
Richard Allen didn't invent the automobile bike rack - his 1967 patent application makes it clear that others came before. But after nearly sixty years selling popular and simple mechanical bike carriers, his company Allen Bikes now offers a line of - yes - Bluetooth-monitored suction cups to stick bikes to your car. If you [ ]
Data centres: Building opportunities on solid foundations
Data centres power New Zealand’s digital economy, enabling cloud , AI and critical services. With billions in investment ahead, collaboration and sustainable infrastructure are key to long-term growth. The backbone of our digital economy Every business-critical system – from banking platforms to supply chains, financial transactions to enterprise applications – relies on data centres. Data centres are the unseen engine rooms: powering cloud platforms, processing expanding AI workloads and underpinning critical services across every industry. The recent NZTech report, Empowering Aotearoa New Zealand’s Digital Future – Our National Data Centre Infrastructure , highlighted the scale of the opportunity for our data centre sector. With 56 operational data centres (four of which are owned and op
Knowledge Map
Connected Articles — Knowledge Graph
This article is connected to other articles through shared AI topics and tags.
More in Products
FinancialContent - Silverback AI Chatbot Introduces AI Assistant Feature to Support Structured Digital Communication and Intelligent Workflow Automation - FinancialContent
FinancialContent - Silverback AI Chatbot Introduces AI Assistant Feature to Support Structured Digital Communication and Intelligent Workflow Automation FinancialContent
Highlights from my conversation about agentic engineering on Lenny's Podcast
I was a guest on Lenny Rachitsky's podcast, in a new episode titled An AI state of the union: We've passed the inflection point, dark factories are coming, and automation timelines . It's available on YouTube , Spotify , and Apple Podcasts . Here are my highlights from our conversation, with relevant links. The November inflection point Software engineers as bellwethers for other information workers Writing code on my phone Responsible vibe coding Dark Factories and StrongDM The bottleneck has moved to testing This stuff is exhausting Interruptions cost a lot less now My ability to estimate software is broken It's tough for people in the middle It's harder to evaluate software The misconception that AI tools are easy Coding agents are useful for security research now OpenClaw Journalists a
Discussion
Sign in to join the discussion
No comments yet — be the first to share your thoughts!