Very unlikely. All the technologies involved work best at scale; for example, the area-to-volume ratio of the liquid gas storage vessel is a critical parameter to keep energy losses low.
I'm probably one of the least educated software engineers on LLMs, so apologies if this is a very naive question. Has anyone done any research into just using words as the tokens rather than (if I understand it correctly) 2-3 characters? I understand there would be limitations with this approach, but maybe the models would be smaller overall?
The way modern tokenizers are constructed is by iteratively doing frequency analysis of arbitrary length sequences using a large corpus. So what you suggested is already the norm, tokens aren't n-grams. Words and any sequence really that is common enough will already be one token only, the less frequent a sequence is the more tokens it needs. That's the Byte-pair encoding algorithm:
Thanks, that's really interesting. Do they correct for spelling mistakes or internationalised spellings? For example, does `colour` and `color` end up in the same token stream?
You will need dictionaries with millions of tokens, which will make models much larger. Also, any word that has too low frequency to appear in the dictionary is now completely unknown to your model.
Along with the other commenter, the reason the dictionary would start getting so big is that words with a stem would have all its variations being different tokens (cat, cats, sit, sitting, etc). Also any out-of-dictionary words or combo words, eg. "cat bed" would not be able to be addressed.
for sure, my point was that usd would already be "better" (more common) than rubles - but yes, 'localized' currencies would be great too (although setting up "adaptive pricing" is a task in itself). baby steps :-)
My point was more about the original comment is fine from the perspective of an American, but for the rest of the world, it doesn’t really matter if it is USD or rubles - it’s still a foreign transaction. I appreciate that for a large percentage of the world, consumers can probably do an approximation of the USD conversion in their head, and not a rubles one, and therefore, USD may be more friendly. That being said, the sales page has already got the approximation in USD anyway, which would be enough for me.
I'd imagine most English-speaking internet users have gotten used to doing local-to-USD conversions. As someone in the US, I usually know about where CAD, AUD, and GBP are relative to me.
Even if you don't know the conversion, something in the range of 50-200% is a lot easier to adjust to, whereas Rubles are on a very different scale (1 GBP = 108 Rubles)
Obviously the ideal would be local listings, but USD is probably the most-familiar reference point if you have to choose exactly one
I have to agree.. given the amount of international business transacted in USD it's a pretty well known currency secondary in most of the world followed by EU then Chinese Yuan and GBP. That said, being in the US can't say how widespread rough translation values of Yuan are to most people outside the Asian/Pacific region.
That implies that all currencies have the same connotation. USD/Pounds/Euros seems much more not scammy to me tha baht or rubles. Especially the latter ones would prevent me from paying in that. Russia is a scam nation.
They are verbose and vague about it: "Some passengers may be concerned about what they can do if they lose their phone or of their devices run out of battery before the pass board the aircraft. Ryanair has said they will assist people experiencing difficulties free of charge at the gate gathering their information and flight details which will be cross-checked and validated against the flight manifest so that they can board as normal."
Of course-- there will be accommodations to start out with. Then, after the new system has become "just the way things work," the accommodations will be removed for security or efficiency or some other reason.
Or maybe not. I've never lost a boarding pass, but if you lose one, you can get it re-issued somewhere, right?
Without endorsement of the behavior, here's a guy getting arrested for being argumentative about not having a boarding pass in the app, and being told he can't pay their 5 dollar boarding pass print fee with cash.
Morally, this guy did nothing wrong. These abusive practices need to be stood up to. I'd rather live in a world where this sort of vehement NO when someone with power tries to pull something unreasonable, like demanding an app or fee, is common and effective, than this world where companies and governments can just steamroll people. Ideally, other passengers would see what's going on, see the systemic problem (even if minor in this case), and also join in in the NO. Make the terminal unusable with an angry growing mass until they decide to be reasonable. The comments on that youtube video mocking the guy and not even addressing the airline's unreasonable policy are also an absurd lack of solidarity against these companies. When someone is mad they probably have a reason to be mad and we ought to listen to them and then get mad too if we agree, other than just dismissing someone for feeling an emotion, lest we'll just have more and more rights, privileges, and respect eroded. We need a culture of standing up for each other against injustice, no matter how small.
I know that his behavior was not a rational pursuit, since in practice humans are too skittish about standing up for themselves and too skittish against anyone whom they see as abnormal/not complying with social norms. But, this does not change the fact that he's completely in the right. I'd love to know a more effective strategy to deal with this shit from companies, if anyone knows one. What should he do instead in this situation where it is simply too unjust to him to be acceptable to give in?
Also, I'm offended at this cop for telling the guy to "be cordial". NO. The airline's behavior is not cordial! They do NOT deserve it back! Freedom of speech means freedom to get mad at someone, possibly REALLY mad, when they try to be unreasonable. Being angry is different from being violent, and the government shouldn't shield people and companies from this consequence (angry people) of their actions.
>The comments on that youtube video mocking the guy and not even addressing the airline's unreasonable policy are also an absurd lack of solidarity against these companies.
I see this a lot on reddit and youtube. I tend to think that it's bots paid for by the company.
There's always just too much unanimous agreement in favor of the corporation.
Maybe I just don't want to believe that people are that homogenized.
The likely future is where you'll be given a USB-C charger to charge your phone. If you have no phone or is broken, it will be the equivalent to having a strongly damaged passport. No fly that day, get a new phone, fly on another date, just like if you needed a new passport. The phone will be your ID, passport, credit card and everything. But since it will be all backed up in Google/Apple/Microsoft cloud, maybe you'll be able to buy a new simple phone near the gate, log in via fingerprint and facial recognition and go on your merry way. But also, once all this stuff is connected up in the cloud, maybe facial and fingerprint recognition will be enough to fly. NFC chips under the skin are probably too bad optics for the near future, but in one or two generations, attitudes will shift.
> I've never lost a boarding pass, but if you lose one, you can get it re-issued somewhere, right?
Yes, typically there's a fee for getting it printed at the check-in counter.
With a normal airline? You walk up to the gate, say you lost your paperwork (boarding pass, ticket, doesn't matter), show your ID, and get a new boarding pass issued within about a minute of managing to get someone's attention. At least that was my experience. No hassle, no fee.
Ryanair? I would expect them to offer you their boarding pass printing service for only $99.99 (you missed the $49.99 special that was only available until 4 hours before boarding, silly you).
Maybe we need to collectively all 'lose' our phone just before boarding at the gate, resulting in some flight delays, so that this nonsense gets reverted.
And the sort of solidarity that unfortunately will never happen. Most people will just download the app and carry on. Very few people hold the very real concern about things moving in this direction.
Your average phone user is already hostage to 7 hours of screentime daily. They don't mind installing more apps. The average person has hundreds of apps on their phone, many of which are never even used.
And they have a constant barrage of notifications, and they tap "Allow" on everything. It can be mind boggling if you spend your life in a developer-minded bubble. They watch ads. They sometimes like the ads and smile at them and don't skip them immediately.
Nothing much has changed since the times when you had to "fix" your aunt's computer in 2003 because it's "slow" and found a zillion toolbars and cleanup/speedup utilities.
The developers of git will continue to be motivated to contribute to it. (This isn’t specific to Rust, but rather the technical choices of OSS probably aren’t generally putting the user at the top of the priority list.)
The UK banned single use plastic bags at major supermarkets. We all moaned about it for a few minutes, forgot our reusable bags a couple of times and then got on with it. Even the small plastic bags you put fruit or pastries in are now gone in a few super markets - initially, they replaced them with transparent paper-based windowed bags, but then I think people realised you really don't need to see inside the bag, and brown paper bags are back.
Yeah, I still don't understand why brown paper bags aren't more standard for everything.
I do see some manufacturers reducing plastic, fortunately. For example, my box of tea bags used to come wrapped in plastic, and now it suddenly doesn't, and I'm wondering why it ever needed plastic. But there's still so much stuff that comes wrapped in plastic, and often multiple layers of it.
Brown paper, from recycled fibers are often contaminated with mineral oil residue (e.g. from ink on paper) and other unhealthy chemicals, sadly.
There was a report in Germany, years ago, of a range of organic products that failed during testing. They discovered the packaging (recycled paper) was the issue, not the crops and the supply chain before packaging.
So, a _really_ biodegradable cellulose bag is desirable. Even if only to use it I side a brown bag (to stabilise it).
Road to hell is paved with good intentions... I wonder how many here even notice the most important comment here from you and just keep repeating how plastic bags are worse.
Yes they are terrible, but we shouldn't just blindly replace them with anything and call it a day but do the (continuous) investigation for best solution, poisons are these days everywhere.
Wouldn't the best solution be ensuring they all end up in an appropriate landfill rather than a river?
It seems people are so against landfills that they're happy to sort their plastic and sent it on an epic journey of fraud where it ends up in a river in India. Meanwhile it could have been buried with their other trash and appropriately managed.
The issue with recycling, as-practiced, is that there's no lifecycle accounting (in many countries, including most of the US).
If we boosted plastic price at point of sale by a recoverable amount, claimable when returning the container for recycling, we'd get higher participation.
Separately, we should also apply the same to the post-return lifecycle: company pays a premium for the material flow, then it rebated that premium upon proof of recycling.
if energy is a problem then surely we'd just build global recycling plants at geothermal hotspots? it's not like shipping is a problem. the sense I get is that the main bottleneck with recycling isn't energy, but labour. handling and sorting rubbish properly is tedious and unpleasant and the west doesn't want to spend the money that its workers would expect for it
tangentially--and I'm aware this sounds incredibly stupid, and I'm sure it is--but on the topic of geothermal hotspots, what is the downside of finding some lava/magma source deep, deep underground and just dumping rubbish in there? surely most of the fumes would just be absorbed before they reach the surface? is it just too expensive of an idea/has it been done/is it likely to have undesirable long term side-effects/do we simply not have safe access to such things
>It seems people are so against landfills that they're happy to sort their plastic and sent it on an epic journey of fraud where it ends up in a river in India
See prior comment about road to hell being paved with good intentions.
About a year or so ago, somebody in the chain of suppliers of plastic PET bottles for seltzer water, used by several different brands, switched to a recycled plastic with a distinct dark tint to it. Immediately obvious because the product, water, is obviously clear.
My family returned six cases of 15 bottles each to Costco, then found that the other brands at local stores were the same way. A couple of months later the bottles went back to normal. I still wonder if they switched back due to customer rejection of the new plastic, or if they found the new plastic was in some way leeching contaminants.
New plastic doesn’t have that problem and is incredibly cheap.
Take price as a proxy for resource / energy input and see that new plastic is also incredibly lite on inputs.
New plastic may have some off-gassing / contact contamination concerns though.
Last time I checked, energetically we’re better off using plastic over paper or recycled plastic, and burying the waste… if we could do that reliably, which we don’t seem to be able to.
One is "People don't like bags stuck in the branches of trees and clogging waterways in their parks". Lightweight plastic shopping bags are so thin that a light breeze can pick them up and loft them up into the air easily. They cost approximately nothing - <2 cents retail, significantly less in bulk. It is incredibly expensive by comparison to pay someone to remove them from tree branches and riparian zones - tens of dollars in wages, equipment, and liability insurance. This is a pragmatic reason why municipalities passed bag taxes or bans. Forcing people to use paper or heavier-weight plastic bags that don't blow in the wind, even if they're not in practice "reusable", solves this one. Taxing them 5 cents or 10 cents or 25 cents per bag nudges a high percentage away.
At some point there are so many bricks in the road, it's direction is so clear, that the intentions are not longer good. At best they are ignorant, but too often they are self serving malice sailing under the flag of ignorance.
I'm old enough to remember when supermarkets only had brown paper bags. They were weak and the handles tear off easily, and anything cold will make the bag wet and it will fall apart usually from the bottom. Supermarkets must have spent a lot of money replacing customer's broken items when bags failed even before leaving the store.
So when doing the calculus for brown paper bags don't forget to include the cost goods wasted when they fail.
Thankfully we did the full stupid circle quickly enough that the gray hairs in the paper bag industry remembered this and the current generation of bags lacks the handles so people are forced to carry them from the bottom.
Those are not the brown paper bags the GP was referring to. Those fall under the earlier category of "forgot our reusable bags a couple of times and then got on with it". The ones that are left are to replace "small plastic bags you put fruit or pastries in".
Australian supermarkets have excelled at replicating this paper bag fiasco.
The white plastic bags they replaced are magnitudes of order more durable and able to carry, I should test this, at a guess ten times the weight. Basically you can fill a white plastic bag with 1.25 litre water bottles to the extent no more can physically fit in the bag and it will be safe to carry and reuse 50 times.
Fortunately the white plastic bags are still available online (eBay / Amazon / etc) so I just buy 50 for my own use as required and use them till they nearly fall apart then repurpose them as bin liners.
They’re incredibly cheap, don’t really get dirty in an unhygienic way, can be washed if something does spill in them, and they fold up in to almost no space.
> Basically you can fill a white plastic bag with 1.25 litre water bottles to the extent no more can physically fit in the bag and it will be safe to carry and reuse 50 times.
Yeah that's not good, the way they do that is with more plastic in the bags. A single bag weighs as much as 5-10 old timey single use bags.
I'm old enough to remember when supermarket brown paper bags didn't have handles... Agree with the other commenter that the handles are pointless but the bags work fine if you just ignore them.
Incidentally, given that I'm _not_ old enough to remember a time before supermarkets had plastic bags, either the invention of attaching handles to paper bags took a very long time to migrate to my corner of California, or this comment makes no sense
I'm old enough to remember when supermarkets had boxes. All the goods they sell comes to them in big cardboard boxes, and and supermarkets would have a fenced off area where they dumped all those boxes, so whenever a customer needed a box to put their groceries in to take them home, they'd get a box from that fenced off area.
I haven't seen those in decades unfortunately. It was a great way to reuse those boxes.
I don't recall ever seeing one of those in the person-facing parts of the store, but I've not had issues either asking someone who works there in the store or going around the back of the store where they do loading and unloading and asking there.
The handles on brown paper bags are noob traps. You're supposed to hold the bag against your body with one arm, your hand on the bottom of the bag. They work fine like this. I've walked home totaling hundreds if not thousands of miles (two or three times a week for many years) with paper grocery bags like this and never had issues.
I think banning plastic completely in packaging is a much harder ask, as whether it is needed is rather nuanced (if I understand it correctly). For example, it's perfectly possible to deliver cucumbers to an end customer without them being shrinkwrapped. However, to deliver enough cucumbers to enough customers for a supermarket scale, I understand from several documentaries that plastic is still required in that case. (For those outside the UK, plastic covered cucumber is the social barometer for plastic packaging.) Banning plastic bags was easy and simple, and our laws don't tend to deal with nuance very well...
Obviously the people who want to buy organic and the people who want to avoid plastic the most are probably almost the same group. They know this. It feels like "Fuck you environmental-aware buyers" to me.
Of course wrapping everything non-organic is a no go as well, it would be terrible for the environment. And I'm afraid stopping the production of non-organic stuff ain't happening anytime soon.
I believe the real solution if possible until they fix this is to go to a market or an organic store where nothing is in plastic, at least for fruits and vegetables.
> Obviously the people who want to buy organic and the people who want to avoid plastic the most are probably almost the same group. They know this. It feels like "Fuck you environmental-aware buyers" to me.
They're different types of environmental. One is "I don't like pesticides and I have money" and the other is "I don't like eternal plastic waste".
Different things, same group of people (money matters aside - people don't buy because it's more expensive, but despite it), no?
The "I have money" part is obviously unfortunate. Buying healthy and environmentally-friendly shouldn't be conditioned by money. The next best individual thing is voting with one's own wallet in the meantime.
The "I don't like pesticides"¹ and the "I don't like eternal plastic waste" are very compatible. Both pesticides and eternal plastic waste hurt the environment in their own ways.
I suppose the target is the restricted set of people who are interested in organic products for their own individual health and who don't push the reasoning far enough to see that their health depends on the environment being healthy in the long term. Or, people who prefer buying organic food and who will make a compromise.
Do you have a different reading?
¹ we will note that organic doesn't mean "no pesticides", and is broader than just pesticides, but I accept the shortcut.
Indeed, but removing the money part of your sentence:
> They're different types of environmental. One is "I don't like pesticides" and the other is "I don't like eternal plastic waste".
Makes its clear that both concerns would come from the same group of people, more or less.
Or not? This is my question to you. Just take my previous comment as "What do you mean, different?".
You have a point with your money thing. Supermarkets absolutely make their choices with individualistic assumptions, taking in account classes of people and their revenues, and I suspect this is how we ended up with this wrapped organic vegetables heresy.
Yeah, the issue is of course that the supermarket sells both kinds of cucumbers, and they need to be able to distinguish between organic and non-organic cucumbers, which is why they wrap one type in plastic. And of course it's better for the environment if that's the type they sell the least of.
So every step makes sense, but the end result looks ridiculous. Maybe they can use paper wrappers instead? Or maybe just settle on one type of cucumber.
The way I understand it, without the wrapping a much larger percentage of cucumbers need to be thrown away before ever being sold, due to spoilage. That's not a win for the environment.
How is this calculated? I know that growing a cucumber has an environmental cost but so does producing plastic, delivering it and then using machines to shrink-wrap every cucumber.
This study, for instance, [1] looks at CO2 emissions. Which may be a somewhat limited view, but the effect is rather large: adding 5 wrappers around a cucumber (4 of which being useless) would result in about the same CO2 usage as adding no wrapper. And that's not even considering spoilage after the cucumber has been bought by a consumer.
This is the problem though, right? It’s not one league table of environmental goodness - there are tradeoffs that as an educated consumer are impose to navigate.
Brown paper bags were the standard grocery store bag up until sometime in the 1980s. The transition to plastic was pushed by environmentalists with a "save the trees" message focused on how many trees were used to make the paper bags.
Not really. It's just my memory of the times. "Save the trees" was a very big thing for a while, in arguments for avoiding paper and cardboard packaging.
A lot of so-called environnemental awareness campaigns are the work of trade organizations or multinationals. Often they shift the blame onto the consumer, so for example it's "you need to recycle" instead of "we need to produce less".
I agree getting rid of plastic bags is a net win for the reasons discussed.
But I can't take the brown paper bag thing seriously! They are a UX nightmare in my workflows. Carry one bag per trip in multiple trips (Instead of ~4 I can do with reusable or plastic). Or try the handled ones which tear off end up with groceries all over. Reusable bags are nice though.
What you see in a lot of places that have people heavily relying on things like delivery services, is people using the reusable bags like they would use single-use bags - so now you have spent even more resources on a bag that's still being used as single-use. Oops.
The way it was explained to me isv that there are so many plastic feedstocks produced by fuel production that it's often most efficient to pay someone just to take it away.
That's basically the economic equivalent of having to pay to get rid of a fallen tree despite that tree then going on to be chipped and sold in bulk to whatever the nearest local place buying chips is.
The feed stock is basically worth nothing, it's the labor and energy investment that you add to it at every step that adds the value.
I'm pretty sure my tea bags are paper, and have always been paper. It's the more recent "pyramid" shaped tea bags that I think are made of plastic. The most recent change to my tea bags was to remove the staple so they could go in organic waste.
I doubt that's made any kind of environmental/ecological impact at all. The cheap, flimsy plastic carrier bags contain orders of magnitude less material than the reusable kind, and had a second life as a bin liner. Now I need to buy bin liners, which are usually made out of sturdier plastic on top of having to get a reusable bag.
Most of the plastic involved in getting food from farm to home isn't the carrier bag or even the food wrappers. It's the massive amount of plastic that pallets of goods are wrapped in for shipping, which happens several times throughout the supply chain.
We should focus on the latter, instead of the former. Pretty much all we're doing is virtue signalling and maybe hoping that it'll make a tiny difference.
Heck, even a marginal improvement in fuel efficiency of trucks delivering to grocery stores would probably do more than these plastic bag shennanigans.
Where I live there are entire fields of crops grown under plastic sheeting, and I do not mean reusable plastic greenhouses, I mean sheeting pegged to the ground. And then the produce is boxed up in plastic, stuck on a palette, wrapped in plastic and delivered to the supermarkets.
Then, when I'm in town I see building projects where the entire building is wrapped in plastic sheeting: eight story buildings wrapped like a parcel in plastic. Even the ground-level hoarding that used to plywood boards is now typically covered in plastic sheeting printed with branding.
And the roadworks: what used to be reusable metal signs and barriers have recently switched to plastic signs and plastic barriers. I get these get battered and broken quickly but at least the steel ones would typically get melted down and reused at their end-of-life. I imagine the plastic ones just end up in landfill or incinerated.
It does kinda make my home recycling efforts seems futile when commercial enterprises are moving in the opposite direction towards more plastic.
There are significantly less palettes being delivered and handled by significantly less people, thus it is far easier to ensure that the plastic used in the delivery process is disposed of properly. Whereas with the abundance of cheap plastic bags that are available on tap to the masses, disposal turns into a mess. I generally agree with you that we should focus on the whole chain and there's lots of easy wins to be had, but decreasing the amount of plastic that gets stuck in trees or otherwise lost in the *environment* is still a good thing.
I fear that this sort of focus on individual actions has made a lot of people rather upset (e.g. the plastic straws debacle) for very little gains. And I worry that it might backfire.
We bring tupperwares when buying groceries for the meat, ham, cheese, fish etc and even if the cynic might say it's just a "feel good" action, well, I still put a lot of plastics in our recycle bin but we halved it since we started doing that (and some other trick). Yes, it definitely feels good.
Here they put a sheet (that they would use anyway) on the scale to separate it physically from the container, and what enters the container doesn't leave it (or goes to the trash).
I've never seen a meat counter employee re-zero the scale after putting a container or wrap on it. I do occasionally see them reading -0.1oz or something initially, presumably to avoid the need for the employees to zero it out each time they weigh something.
> The UK banned single use plastic bags at major supermarkets. We all moaned about it for a few minutes, forgot our reusable bags a couple of times and then got on with it.
I hope you're right. Here in Norway, the sensible people did what you describe. A large minority has, on the other hand, turned the lack of plastic bags (and straws, which I'm sure they barely even used once in a blue moon before) into a battlefield of the culture wars. And far right politicans of course cater to them. They manage to capture discourse talking about "environmentalism gone wild" and "EU overreach". It's terribly annoying and they manage to waste everyone's time and derail important debates with this nonsense.
Funnily you can still buy packs of plastic straws, just that they are sold as "resuable" with a cleaning brush (which noone likely ever uses). They are simply not the default option now and that's enough to make some people rage.
These days, you never hear about reduce, reuse, recycle, and how its supposed to be in that priority order. When i was a kid thats what we were taught. Now its just recycle, recycle, recycle
My conspiracy theory is corporate propaganda changed it because reduce and reuse decreases demand, while recycle potentially only lowers production cost
I highly recommend the documentary Plastic Wars (Frontline). It’s about how the plastics industry made a major marketing push for recycling starting in the 80s, in order to avoid plastic bans and ensure production continued to increase. It shifted the burden of plastic waste from producers to consumers, and we are essentially still in that conceptual space (at least in the US).
For sure. Plastic packaging keeps the product fresh and hermetically sealed from the clean factory / production depot to your store and eventual home. Get rid of plastic and there will be a LOT more spoilage.
Maybe that's an acceptable tradeoff, but most people don't even realize there is a tradeoff being made...
Since all our local markets have introduced handheld scanners, I don't even bring my bags in. I put everything in the cart barcode up, get to the checkout, scan everything, pay, and go.
When I get to the car I unload into the bags. I'm sure it's not a thing for everyone, but I feel like I'm cutting out a fair bit of shuffling.
I get the impression that the 'handheld scanner' may be tethered to the till (like in B&Q) rather than one you can carry round with you (like Sainsbury's/Asda/Tesco)
For most you don't. But if you buy loose cherry tomatoes, having 30 of them rolling around everywhere isn't exactly practical.
Thats easily solved though by simple buying some reusable fruit/veggie net. Essentially the same as what you would use for socks or underwear in the laundry.
This is a really unhelpful attitude. There are periods of life where buying doesn't make sense and it's financially impossible - landlords provide people with a home at these times. I'm not saying there aren't bad landlords - there are - but being a landlord isn't inherently bad; they are providing an essential service for society.
Providing a place to live is surely not one of those services though. There will always been some portion of the population that can't afford to buy a home. Without landlords, what are those people supposed to do?
>Providing a place to live is surely not one of those services though
that's not what landlords are. That's the construction company or the building manager and he or she's indeed doing a great service. Landlords are absentee owners who extract economic rent. You can of course, like say Vienna, nationalize most of the housing stock and hire people who provide actual services just the same.
I'm not sure how, and maybe some of the coding agents are doing this, but we need to teach the AI to use abstractions, rather than the whole code base for context. We as humans don't hold the whole codebase in our hear, and we shouldn't expect the AI to either.
They already do, or at least Claude Code does. It will search for a method name, then only load a chunk of that file to get the method signature, for example.
It will use the general information you give it to make educated guesses of where things are. If it knows the code is Vue based and it has to do something with "users", it might seach for "src/*/User.vue.
This is also the reason why the quality of your code makes such a large difference. The more consistent the naming of files and classes, the better the AI is at finding them.
LLMs (current implementation) are probabilistic so it really needs the actual code to predict the most likely next tokens. Now loading the whole code base can be a problem in itself, since other files may negatively affect the next token.
Sorry -- I keep seeing this being used but I'm not entirely sure how it differs from most of human thinking. Most human 'reasoning' is probabilistic as well and we rely on 'associative' networks to ingest information. In a similar manner - LLMs use association as well -- and not only that, but they are capable of figuring out patterns based on examples (just like humans are) -- read this paper for context: https://arxiv.org/pdf/2005.14165. In other words, they are capable of grokking patterns from simple data (just like humans are). I've given various LLMs my requirements and they produced working solutions for me by simply 1) including all of the requirements in my prompt and 2) asking them to think through and 'reason' through their suggestions and the products have always been superior to what most humans have produced. The 'LLMs are probabilistic predictors' comments though keep appearing on threads and I'm not quite sure I understand them -- yes, LLMs don't have 'human context' i.e. data needed to understand human beings since they have not directly been fed in human experiences, but for the most part -- LLMs are not simple 'statistical predictors' as everyone brands them to be. You can see a thorough write-up I did of what GPT is / was here if you're interested: https://photonlines.substack.com/p/intuitive-and-visual-guid...
I'm not sure if I would say human reasoning is 'probabilistic' unless you are taking a very far step back and saying based on how the person lived, they have ingrained biases (weights) that dictates how they reason. I don't know if LLMs have a built in scepticism like humans do, that plays a significant role in reasoning.
Regardless if you believe LLMs are probabilistic or not, I think what we are both saying is context is king and what it (LLM) says is dictated by the context (either through training) or introduced by the user.
'I don't know if LLMs have a built in scepticism like humans do' - humans don't have an 'in built skepticism' -- we learn in through experience and through being taught how to 'reason' within school (and it takes a very long time to do this). You believe that this is in-grained but you may have forgotten having to slog through most of how the world works and being tested when you went to school and when your parents taught you these things. On the context component: yes, context is vitally important (just as it is with humans) -- you can't produce a great solution unless you understand the 'why' behind it and how the current solution works so I 100% agree with that.
For me, the way humans finish each other's sentences and often think of quotes from the same movies at the same time in conversation (when there is no clear reason for that quote to be a part of the conversation), indicates that there is a probabilistic element to human thinking.
Is it entirely probabilistic? I don't think so. But, it does seem that a chunk of our speech generation and processing is similar to LLMs. (e.g. given the words I've heard so far, my brain is guessing words x y z should come next.)
I feel like the conscious, executive mind humans have exercises some active control over our underlying probabilistic element. And LLMs lack the conscious executive.
e.g. They have our probabilistic capabilities, without some additional governing layer that humans have.
I think the better way to look at it is that probabilistic models seem to be an accurate model for human thought. We don't really know how humans think, but we know that they probably aren't violating information theoretic principles, and we observe similar phenomena when we compare humans with LLMs.
Yep, just like like looking at a birds feather through a microscope explains the principles of flight…
Complexity theory doesn’t have a mathematics (yet), but that doesn’t mean we can’t see that it exists. Studying the brain at the lowest levels haven’t lead to any major insights in how cognition functions.
I personally believe that quantum effects play a role and we’ll learn more once we understand the brain at that level, but I recognize that is an intuition and may well be wrong.
You seem possibly more knowledgeable then me on the matter.
My impression is that LLMs predict the next token based on the prior context. They do that by having learned a probability distribution from tokens -> next-token.
Then as I understand, the models are never reasoning about the problem, but always about what the next token should be given the context.
The chain of thought is just rewarding them so that the next token isn't predicting the token of the final answer directly, but instead predicting the token of the reasoning to the solution.
Since human language in the dataset contains text that describes many concepts and offers many solutions to problems. It turns out that predicting the text that describes the solution to a problem often ends up being the correct solution to the problem. That this was true was kind of a lucky accident and is where all the "intelligence" comes from.
So - in the pre-training step you are right -- they are simple 'statistical' predictors but there are more steps involved in their training which turn them from simple predictors to being able to capture patterns and reason -- I tried to come up with an intuitive overview of how they do this in the write-up and I'm not sure I can give you a simple explanation here, but I would recommend you play around with Deep-Seek and other more advanced 'reasoning' or 'chain-of-reason' models and ask them to perform tasks for you: they are not simply statistically combining information together. Many times they are able to reason through and come up with extremely advanced working solutions. To me this indicates that they are not 'accidently' stumbling upon solutions based on statistics -- they actually are able to 'understand' what you are asking them to do and to produce valid results.
If you observe the failure modes of current models, you see that they fail in ways that align with probabilistic token prediction.
I don't mean that the textual prediction is simple, it's very advanced and it learns all kinds of relationships, patterns and so on.
But it doesn't have a real model and thinking process relating to the the actual problem. It thinks about what text could describe a solution that is linguistically and language semantically probable.
Since human language embedds so many of the logics and ground truths that's good enough to result in a textual description that approximate or nails the actual underlying problem.
And this is why we see them being able to solve quite advanced problems.
I admit that people are wondering now, what's different about human thinking? Maybe we do the same, you invent a probable sounding answer and then check if it was correct, rinse and repeat until you find one that works.
But this in itself is a big conjecture. We don't really know how human thinking works. We've found a method that works well for computers and now we wonder if maybe we're just the same but scaled even higher or with slight modifications.
I've heard from ML experts though that they don't think so. Most seem to believe different architecture will be needed, world models, model ensembles with various specialized models with different architecture working together, etc. That LLMs fundamentaly are kind of limited by their nature as next token predictors.
I think the intuitive leap (or at least, what I believe) is that meaning is encoded in the media. A given context and input encodes a particular meaning that the model is able to map to an output, and because the output is also in the same medium (tokens, text), it also has meaning. Even reasoning can fit in with this, because the model generates additional meaningful context that allows it to better map to an output.
How you find the function that does the mapping probably doesn't matter. We use probability theory and information theory, because they're the best tools for the job, but there's nothing to say you couldn't handcraft it from scratch if you were some transcendent creature.
The text of human natural language that it is trained on encodes the solutions to many problems as well as a lot of ground truths.
The way I think of it is. First you have a random text generator. This generative "model" in theory can find the solution to all problems that text can describe.
If you had a way to assert if it found the correct solution, you could run it and eventually it would generate the text that describes the working solution.
Obviously inefficient and not practical.
What if you made it so it skipped generating all text that aren't valid sensical English?
Well now it would find the correct solution in way less iterations, but still too slow.
What if it generated only text that made sense to follow the context of the question?
Now you might start to see it 100-shot, 10-shot, maybe even 1-shot some problems.
What if you tuned that to the max? Well you get our current crop of LLMs.
What else can you do to make it better?
Tune the dataset, remove text that describe wrong answers to prior context so it learns not to generate those. Add more quality answers to prior context, add more problems/solutions, etc.
Instead of generating the answer to a mathematical equation the above way, generate the Python code to run to get the answer.
Instead of generating the answer to questions about current real world events/facts (like the weather). Have it generate the web search query to find it.
If you're asking a more complex question, instead of generating the answer directly, have it generate smaller logical steps towards the answer.
But documentation doesn't include styling or preferred pattern, which is why I think a lot people complain that the LLM will just produce garbage. Also documentation is not guaranteed to be correct or up to date. To be able to produce the best code based on what you are hoping for, I do think having the actual code is necessary unless styling/design patterns are not important, then yes documentation will be suffice, provided they are accurate and up to date.
the fact we cant keep the repo in our working memory is a flaw of our brains. i cant see how you could possibly make the argument that if you were somehow able to keep the entire codebase in your head that it would be a disadvantage.
Information tradeoff. Even if you could keep the entire code base in memory, if something else has to be left out of memory, then you have to consider the value of an abstraction verses whatever other information is lost. Abstractions also apply to the business domain and works the same.
You also have time tradeoffs. Like time to access memory and time to process that memory to achieve some outcome.
There is also quality. If you can keep the entire code base in memory but with some chance of confusion, while abstractions will allow less chance of confusion, then the tradeoff of abstractions might be worth it still.
Even if we assume a memory that has no limits, can access and process all information at constant speed, and no quality loss, there is still communication limitations to worry about. Energy consumption is yet another.
I think they're recognising patterns, which is not the same thing.
Abstractions are stable, they're explicit in their domains, good abstractions cross multiple domains, and they typically come with a symbolic algebra of available operations.
Math is made of abstractions.
Patterns are a weaker form of cognition. They're implicit, heavily context-dependent, and there's no algebra. You have to poke at them crudely in the hope you can make them do something useful.
Using LLMs feels more like the latter than the former.
If LLMs were generating true abstractions they'd be finding meta-descriptions for code and language and making them accessible directly.
AGI - or ASI - may be be able to do that some day, but it's not doing that now.
Compact is a reasonable default way to do that, but quite often it discards important details. It's better to have CC to store important details, decisions and reasons in a document where it can be reviewed and modified if needed.
There is a category called "Annie Maunder Open Category", which allows for creative use of generative AI, as long as it is declared, otherwise it is banned. You have to take that slightly with a pinch of salt though, as many of the images don't represent what could be seen with the naked eye. It could be as simple as a wide spectrum sensor (which I don't think anyone could claim is AI) through to alignment of many 100s of images (for which the algorithms in use may have been considered AI a decade or 2 ago).
reply