There's plenty of evidence that good prompts (prompt engineering, tuning) can result in better outputs.
Improving LLM output through better inputs is neither an illusion, nor as easy as learning how to google (entire companies are being built around improving llm outputs and measuring that improvement)
Sure, but tricks & techniques that work with one model often don't translate or are actively harmful with others. Especially when you compare models from today and 6 or more months ago.
Keep in mind that the first reasoning model (o1) was released less than 8 months ago and Claude Code was released less than 6 months ago.
We think Grafana is still a great tool and many teams are heavily invested in the Grafana ecosystem. We'll continue to invest in Grafana support via ClickHouse's official Grafana plugin and that won't be changing at all with this release.
However, there's a bit of a fundamental difference in the user experience we're targeting. Grafana has really excelled at traditional monitoring dashboards, low cardinality monitoring workflows.
ClickHouse unlocks a newer paradigm of high cardinality, high performance observability. It enables a new set of workflows/UX that allows engineers to query novel problems quickly as opposed to working off of static dashboards. That's really a big focus of ours, so you'll see we do exploration/search/syntax/UI layout is quite different from Grafana due to this.
At this point it isn't even an original realization of ours. Just as an example, Shopify built a complete custom app (only keeping the auth part of Grafana) while migrating to ClickHouse for similar reasons.
Ah sorry I missed that part of the question, yes MongoDB and ClickHouse are the two stateful services. We'll be looking to see if we can offer some mode to simplify it down to just ClickHouse but that'll take a bit more work.
Totally agree - you use an observability tool because it answers your questions quickly, not just return searches quickly.
Beyond raw performance and cost effectiveness, which is quite important at scale, we work a lot on making sure the application layer itself is intuitive to use. You can always play around with what ours looks like at play.hyperdx.io :)
Otherwise yes you can authenticate against the other versions with a email/password (really the email doesn't do anything in the open source distribution, just a user identifier but we keep there to be consistent)
In theory you should be able to try using FerretDB for example.
We have this on the medium term roadmap to investigate proper support for a compatibilty layer such as ferret or more likely just using ClickHouse itself as the operational data store.
Hey, I'm a maintainer of HyperDX. I'd love to chat with you about a potential collaboration. we're planning to migrate off mongodb. Please reach out to me on Discord (warren)
Luckily ClickHouse and serious throughput are pretty synonymous. Internally we're at 100+PB of telemetry stored in our own monitoring system.
Vector supports directly writing into ClickHouse - several companies use this at scale (iirc Anthropic does exactly this, they spoke about this recently at our user conference).
Please give it a try and let us know how it goes! Happy to help :)
Thanks! Very familiar with ClickHouse, but can logs then be ingested into CH without going through HyperDX? Doesn't HyperDX require a specific schema that the Vector pipeline would have to adapt the payloads to?
Nope! We're virtually schema agnostic, you can map your custom schema to observability concepts (ex. the SQL expression for TraceID, either a column or a full function/expression will work).
We don't have any lock in to our ingestion pipeline or schema. Of course we optimize a lot for the OTel path, but it works perfectly fine without it too.
Very happy with CH, but I'm sadly disappointed with HyperDX:
* Not possible to reorder columns?
* Not possible to wrap cells?
* The doesn't seem to be a concept of "field popularity" as Kibana has (where you can also "pin" fields)?
* The log view's chart is very simplistic. No breakdown. Time selector is very primitive. Look at Google Cloud's log view if you want to see something good here.
* No field value autocompletion when using the query builder?
* Live view is annoying as hell. It scrolls to the top every few seconds even when there is no need data.
* Chart view is nearly useless. I tried to create a chart showing two time series calculations, average and P95 of a metric, and it doesn't draw it correctly, and the series get messed up, and overall I think it's not usable. Happy to explain further, but even a cursory test should reproduce this. Fortunately Grafana can access CH data and do a much better job here.
* The drill-down sidebar doesn't seem to have any idea what's important or not, so some fields I will never want to filter on are high up in the list and others are further down. I can't rearrange the fields?
* Lots of other nuisances.
Overall I found it much, much weaker than Kibana for both logs and charts, and Kibana is already pretty atrocious at both, so.
I can probably live with it because I'm desperate to replace Elastic with CH, but I think I will kiss the functionality of the Kibana UI.
- Reorder columns: You should be able to do so by modifying the order of the SELECT statement at the top.
- Wrapping: totally agree - this is something we're adding in, it's on the near term todos.
- Pin fields: you're totally right, you can pin values but that doesn't prioritize the filter in general. I'm getting this one in the queue.
- Time picker: I hear you with Google Cloud's time picker, they did indeed make a really nice one. Though I'm curious to hear more about the breakdown, is it wanting to customize how that chart is grouped beyond log level?
- Autocomplete: We had a regression if you're on v2.0.0 which is what I suspect you're hitting. If you're on v2.0.1 lucene should have field value autocomplete. We don't have it yet for SQL.
- Live View: Can you clarify what is scrolling? Or is this about disabling live?
- Chart View: I just tried really quick and it seems okay to me, do you mind sharing more details?
- Sidebar: Is this related to field popularity to pin fields or something else?
This feedback is all super helpful - you can probably tell we're still early in building out the perfect experience. I'd love it if we could dive in deeper either on our discord (https://hyperdx.io/discord) or email: michael.shi@clickhouse.com. Either way this has been exceptionally helpful :)
Thanks for the reply. I'll probably show up on Discord eventually!
- Reordering is something that should be possible in the UI, not just by rewriting the query. Typically when I look at logs, I rapidly iterate on my view, adding columns, removing columns, rearranging them so I can see certain things side by side. This process should be zippy and friction-free.
- Breakdown: Here is an example from Kibana: https://imgur.com/Dkz6j9K. You can make the timeline display a stacked bar chart by any dimension.
- Autocomplete:
- Live view: Maybe I ran into a bug when I ran it locally; if I enabled the live tail mode and then scrolled down, then after about 2 seconds the whole page would scroll to the top even though nothing new had come in. When I try your live demo now, I see that it disables the live view when I scroll. I would argue that this is also bad UX, as it requires me to click "Resume" again. It should at least resume the mode if I scroll back up.
- Chart view: Maybe I was running into another bug that got fixed. I started latest HyperDX in Docker Compose and can't reproduce it now.
I did run into more chart bugs: Drag-to-zoom causes text to be selected, and when I zoom, the X axis range doesn't exactly match what I zoomed into, I.e. if I select from the start to the end of a curve, I expect the zoomed-in view to start and end at the same place, unless something funky is being done to the rounding of each X point. I have a video here: https://imgur.com/p6XOZ6U. [Edit: It is about granularity. I see the zoom is correct, it's just that your auto-granularity is very aggressive: https://imgur.com/C3TarFo. I've never seen this with Grafana, which bases its calculated time interval on the number of horizontally viewable data points, i.e. how many pixels wide the chart view is. Maybe an idea you should steal?]
For some reason the chart is working better for me now, but I would still argue that you're miles and miles behind Kibana. Kibana has a number of visualizations (line, bar, stacked bar, area, table, etc.; line smoothing of the kind you do is often not desirable, and a staircase is often better), and supports different aggregations such as group by top values: One thing I do constantly is to break a chart down by top K values. For example, a stacked bar chart of sum(duration) grouped by customer ID can be a very nice visualization.
I do like how HyperDX has a filter for every data series, which Kibana hides behind a convoluted UI.
- Sidebar: Yes, I suppose so!
Another thing I thought of was the filters in the sidebar. Seeing "ResourceAttributes['foo']" isn't so nice UX, when most of the fields are resource attributes, and they typically get truncated horizontally. From your demo: https://imgur.com/a/69lD3t5.
Echoing the comment below, I guess one obvious thing is that we are a team at ClickHouse and an official first-party product on top. That translates into:
- We're flexible on top of any ClickHouse instance, you can use virtually any schema in ClickHouse and things will still work. Custom schemas are pretty important for either tuned high performance or once you're at a scale like Anthropic. This makes it also incredibly easy to get started (especially if you already have data in ClickHouse).
- The above also means you don't need to buy into OTel. I love OTel but some companies choose to use Vector, Cribl, S3, a custom writing script, etc for good reasons. All of that is supported natively due to the various ClickHouse integrations, and naturally means you can use ClickStack/HyperDX in that scenario as well.
- We also have some cool tools around wrangling telemetry at scale, from Event Deltas (high cardinality correlation between slow spans and normal spans to root cause issues) to Event Patterns (clustering similar logs or spans together automatically with ML) - all of these help users dive into their data in easier ways than just searching & charting.
- We also have session replay capability - to truly unify everything from click to infra metrics.
We're built to work at the 100PB+ scale we run internally here for monitoring ClickHouse Cloud, but flexible enough to pin point specific user issues that get brought up once in a support case in an end-to-end manner.
There's probably a lot more I'm missing. Ultimately from a product philosophy standpoint, we aren't big believers in the "3 pillars" concept, which tends to manifest as 3 silos/tabs for "logs", "metrics", "traces" (this isn't just Signoz - but across the industry). I'm a big believer that we're building tools to unify and centralize signals/clues in one place and giving the right datapoint at the right time to the engineer. During an incident I just think about what's the next clue I can get to root cause an issue, not if I'm in the logging product or the tracing product.
So, seems like the direction you are going is trying to enable ingestion to different ClickHouse instances (Cloud/BYOC/Self hosted) and then use HyperDX as the query & visualization layer on top.
I think, fundamental difference we have at SigNoz on how we approach this is that we want to solve for observability and the fact that we use ClickHouse today is just a point in time fact. In future, we are open to use any other datastore which may be more performant for observability. We can also use different databases to augment different use cases in observability.
>Ultimately from a product philosophy standpoint, we aren't big believers in the "3 pillars" concept, which tends to manifest as 3 silos/tabs for "logs", "metrics", "traces" (this isn't just SigNoz - but across the industry).
I am not too sure on how this works in practice, do you expect people to write metrics and logs query in the same explorer. From our experience, the query writing experience is very different for logs and metrics and you need different defaults to make the query writing UX easier for users.
Improving LLM output through better inputs is neither an illusion, nor as easy as learning how to google (entire companies are being built around improving llm outputs and measuring that improvement)