Add log entry-specific retention dates

[jongpie]

Discussed in #689

^{Originally posted by josegonzalezNorthius May 22, 2024}
Hey,

We recently introduced NebulaLogger in our organization and have integrated it into many different areas. While monitoring an integration, I activated the logs at the info level, but I would like to delete the info level logs earlier than the error level logs. For this reason, I propose the following change. Let me know if this could be a good idea.

Custom Retention by Log Level: Users will be able to define specific retention periods for each log level (e.g., debug, info, warning, error). This ensures that critical logs can be preserved longer while less important logs can be discarded more quickly.

A solution could be to add separate retention fields for each log level instead of a single retention period for all logs.

[jhawk-4]

We are just in the beginning stages of implementing Nebula, but I could also see the benefit of being able to control retention by tags. In this way, we can tag items more specific to compliance that may require longer evidence. For those items we could apply a tag that has an associated retention policy with it. Then all of the log items across any logging level would get that policy applied to them. I also understand that this could be tricky logically trying to determine the right retention when multiple tags get applied.

[jongpie]

@jhawk-4 that's not a bad idea - I think if there were multiple tags with different retention dates, it would make the most sense to use the longest retention date. For example, if a log entry has 2 tags:

1. Tag A: retention of 30 days
2. Tag B: retention of 50 days

Then 50 days would be used. Let me know if that approach makes sense to you.

And if tags are going to have a retention date, then I'd probably go ahead and include a way to control it for logger scenarios (LoggerScenario__c) too. I'll have to think about all of this more, as the scope & complexity grows quickly - but I think the end result would provide a lot of controls for retention dates.

[jhawk-4]

Agreed that the max date makes the most sense to win.

And given that I just dropped in and increased the scope, it would be polite of me to offer to help. So, if I can help out here, please let me know.

[jongpie]

@jhawk-4 that's incredibly kind of you to offer - I'm currently trying to wrap up a few other bugfixes & releases over the next few weeks, so I'm not sure when exactly I'll look into this enhancement more. But whenever I do, I'll definitely let you know if I need some help. (And your feedback has already been helpful, thanks for the input!)