Skip to main content
Question

Would this be a max active requests settings issue?

  • March 11, 2025
  • 18 replies
  • 62 views

We're seeing a lot of these 500 errors to our HEC endpoint from Stream. And customers are reporting receiving 503 status codes. Would this be a max active requests settings issue? Maybe something else?

18 replies

  • Author
  • Employee
  • March 11, 2025

If this turns out to be the issue, this parameter can be modified to allow more concurrent connections.


  • Author
  • Employee
  • March 11, 2025

The HEC source will return a 503 if there are too many open connections. I'd recommend turning debug for the source to get more detail. This message is logged if w are closing because of too many connections:```this.dLogger.debug('Dropping request because max active connections has been reached', {```


  • Author
  • Employee
  • March 11, 2025

We are thinking it may be the 'max active requests' but im unsure what the repercussions of raising that are. Would it be server performance? Would it be smarter to raise that up or to add more workers to the group?


  • Author
  • Employee
  • March 11, 2025

I dont see where to turn on debug for the source. Is that one of the options under settings>logging?levels ?


  • Author
  • Employee
  • March 11, 2025

To enable debug go to Group Settings -> Logging -> Levels and search for the source there.


  • Author
  • Employee
  • March 11, 2025

Then commit and deploy the changes.


  • Author
  • Employee
  • March 11, 2025

> Would it be server performance? Would it be smarter to raise that up or to add more workers to the group?I think the answer depends on how the current workers are performing, if they are over taxed then adding more workers would make sense. I'll let <@U012ZP93EER&gt; <@U01C35EMQ01&gt; or someone else chime in here.


  • Author
  • Employee
  • March 11, 2025

ahh those settings, ok thanks.


  • Employee
  • March 11, 2025

You might want to consider this setting, if you are sending high volumes:


Repercussions would be additional data being processed. How much? Depends on how much is rejected vs what's being accepted. So if you don't have any spare capacity then you may need to add a bit more if the amount of rejected requests is a non-trivial percentage of the total hitting the nodes. The limit is a throttling mechanism so depending on how much extra capacity was provisioned on your nodes will dictate whether that application level limit can be raised w/o changing anything else.


  • Employee
  • March 11, 2025

Interested to know the use case for sending to the `raw` endpoint vs `event` as the latter batches and is more performant. :slightly_smiling_face:


  • Author
  • Employee
  • March 11, 2025

<@UUP82DJHE&gt; does the full debug message list the number of active connections?


  • Author
  • Employee
  • March 11, 2025

Im wondering if there is a way to monitor and alert when we get close to the limit.


  • Author
  • Employee
  • March 11, 2025

<@U04NPSXKJKV&gt; been set like this for a while. We think we must have tried the event endpoint initially and it wasnt working but dont recall. We're going to test in dev if event will work or has any issues.


  • Author
  • Employee
  • March 11, 2025

Yes, here's the full log message:```debug('Dropping request because max active connections has been reached', { numActive: numInProgressConn, maxAllowed: numMaxActiveReq, statusCode: 503,});```


If the message is logged it means you hit the limit for that worker process configured for the worker group.


  • Author
  • Employee
  • March 11, 2025

ok so it says its been reached, but there isnt an informational saying heres the current amount being used.


the `numMaxActiveReq` is a variable that contains an integer representing the current max