Spring Session Redis Why How and Production Pitfalls

Spring Session Redis
Why, How and Production Considerations
March 2018
Oded Shopen
About Me
Oded Shopen ▪
Software Architect @ Amdocs Israel ▪
Working on Event-Driven, Cloud Native Microservices ▪
🤓 Redis Geek ▪
▪ http://odedia.org
▪ @odedia on twitter
 Information Security Level 2 – Sensitive

2 © 2017 – Proprietary & Confidential Information of Amdocs
Why?

Why Spring Session?
Once upon a time, there was the servlet container. And it was good… ▪
Tomcat 🍪 123-456 = David

Why Spring Session?
🍪 789-111 = Larry 🍪 123-456 = David
Load Balancer
Tomcat Tomcat Tomcat

Why Spring Session?
🍪 789-111 = Larry 🍪 123-456 = ?
Load Balancer
Tomcat Tomcat

Why Spring Session?

Why Spring Session?
Load Balancer
Tomcat
Tomcat
Tomcat
Tomcat Tomcat
Tomcat Tomcat Tomcat Tomcat

Why Spring Session?
Load Balancer
Tomcat
Tomcat
Jetty Tomcat Tomcat
Tomcat
Tomcat Tomcat
Jetty Jetty Tomcat Tomcat

Why Spring Session?
Tomcat
Session Data

Why Spring Session?
Tomcat Tomcat
Jetty
Tomcat
Tomcat
Tomcat
Tomcat
Session Data

Why Spring Session?
▪ Replaces the built-in HttpSession with another implementation
▪ Transparent drop-in replacement when using Spring Boot
▪ Makes your servers truly stateless
▪ Sessions survive application restarts
▪ No need for Load Balancer sticky sessions
▪ Conforms to the cloud-native apps 12-factor principals

Twelve-factor processes are stateless
and share-nothing.
Any data that needs to persist must be

stored in a stateful backing service,
typically a database.

Why Spring Session Redis?
https://docs.spring.io/spring-session/docs/1.3.1.RELEASE/reference/html5/

Why Spring Session Redis?
▪ The application requires frequent, fast access to the session
▪ A fast database is critical
▪ Redis is really fast
▪ No user experience degradation by externalizing the session to Redis
▪ Sessions needs to expire after some time
▪ Redis expiring keys are a great solution
▪ With sharding and clustering, Redis scales when your user base scales

How?

Demo
https://github.com/odedia/spring-session-redis-sample

Production
Considerations

#1 Be Prepared to Scale
▪At the very minimum, use a master/slave setup

#1 Be Prepared to Scale
▪ Consider sharding your sessions in a Redis Cluster
▪ RedisLabs + DNS Proxy will hide the cluster topology and let you focus on
a simple configuration on the client side
▪ No need to configure sentinels
Node 1
spring.redis.sentinel.master=MasterNode
spring.redis.url=proxy:6379 Proxy
Node 2
spring.redis.sentinel.nodes=Node1:6379,Node2:6379,Node3:6379
Node 3
#2 Pool Settings
▪ Spring Data Redis uses a JedisConnectionFactory with the following defaults:
spring.redis.pool.max-active=8
spring.redis.pool.max-idle=8
spring.redis.pool.max-wait=-1
spring.redis.pool.min-idle=0
spring.redis.timeout=0
▪ Behind the scenes - an Apache Commons GenericObjectPool.
▪ Use max-active based on your tomcat/jetty threads.
▪ Use max-wait based on your setup
▪ Too high - all tomcat threads can get stuck, waiting
▪ Too low - your consumers may get errors
▪ Use min-idle of at least 8 to offset “sudden load” issues (such as mass login on a business day).
▪ Set spring.redis.timeout to a reasonably low number (such as 1000ms)
▪ Too high - your connection pool can become full quickly.
▪ Too low - your consumers may get errors

#3 Server-Side Metrics
▪ Spring Boot Actuator provides a production-ready monitoring metrics.
▪ Simply add spring-boot-starter-actuator to your gradle/maven dependencies
▪ /metrics endpoint provides valuable monitoring data
▪ However, Spring Data Redis is not available…
▪ Luckily, actuator is extensible!
▪ https://github.com/nysd/spring-boot-redis-metrics

#4 Monitor Redis
▪ Redislabs monitoring dashboards provide
excellent visibility to identify issues, as we’ll soon
see.
▪ If you use open source and monitor yourself  
on-prem, keep in mind that Redis is a single-
threaded database.
▪ A 2-core virtual machine would report 50%
CPU utilization
▪ However, redis itself might be at 100% at that
time
▪ Monitor the process, not the VM

#5 Please Monitor Responsibly
▪ Be careful how you monitor…
▪ KEYS * runs over all the keys in the DB. Your DB is unresponsive until operation is over!
▪ SMEMBERS returns all keys in a given set. Some sets may contain all keys in the system.
▪ In our use case, we kept track of all logins from a certain vendor, which is pretty
much O(n) cardinality, so same as KEYS command
▪ SCARD simply returns the size of a given set with cardinality of O(1), so it can be used
safely in production
▪ SCARD “spring:session:index:org.springframework.session.FindByIndexNameSessionRepository.PRINCIPAL_NAME_INDEX_NAME:UI_USERS"
▪ INFO provides valuable information (such as instantaneous_ops_per_sec) and can be

safely used in production.

#6 Inside Spring Session Redis
▪ What happens when your system is completely idle (0 users)?
▪ You would expect redis to be idle as well
▪ However…

▪ Multiple logins using:
▪ for ((i=1;i<=10000000;i++));  
do curl -s -u gateway:password localhost:8080/user >/dev/null;
done
▪ One would expect consistent load on  
redis, but in reality, we got:


▪ Time to dig into the code… (thanks, open source!)
▪ Inside RedisOperationsSessionRepository: 
@Scheduled(cron = "${spring.session.cleanup.cron.expression:0 * * * * *}")

public void cleanupExpiredSessions() {
this.expirationPolicy.cleanExpiredSessions();
}
▪ cleanupExpiredSessions() would loop over expired keys, delete them, and “touch”
them to make sure they are immediately deleted.
▪ This means that by default, every minute on the minute, Spring will call delete on every
expired key in Redis.

▪ Why? Redis can self-expire keys just fine, thank you very much
▪ The main reason:
▪ Spring wants to conduct cleanup activities when a session is expired.
▪ The API exposes SessionDeletedEvent and SessionExpiredEvent to let the
program cleanup a user’s session (close web sockets, notify SSO server etc.)
▪ Spring needs the session details available when the cleanup activity is
performed

▪ To solve this issue, two separate keys are managed for each session
▪ An “expiration notification” key expires after 30 minutes
▪ The actual session details key expires 5 minutes later
HMSET spring:session:sessions:33fdd1b6-b496-4b33-9f7d-df96679d32fe creationTime
1404360000000 \
maxInactiveInterval 1800 \
lastAccessedTime 1404360000000 \
sessionAttr:attrName someAttrValue \
sessionAttr2:attrName someAttrValue2
EXPIRE spring:session:sessions:33fdd1b6-b496-4b33-9f7d-df96679d32fe 2100
APPEND spring:session:sessions:expires:33fdd1b6-b496-4b33-9f7d-df96679d32fe ""
EXPIRE spring:session:sessions:expires:33fdd1b6-b496-4b33-9f7d-df96679d32fe 1800
SADD spring:session:expirations:1439245080000 expires:33fdd1b6-b496-4b33-9f7d-
df96679d32fe
EXPIRE spring:session:expirations1439245080000 2100

▪Redis guarantees expired keys would be passively cleaned, but makes no
guarantees on when
▪Spring Session Redis wants to make sure the “expire” key is expired within the 5
minutes window.
▪For this reason, the cronjob updates the expired keys every minute
▪But, there is a risk here…

Tomcat Tomcat
Jetty
Tomcat
Tomcat
Tomcat
Tomcat
Session Data

Tomcat Tomcat
Jetty
Tomcat
Tomcat
Tomcat
Tomcat
Session Data

▪ To recap:
▪ Every minute, each server tries to delete all “expires” keys.
▪ A lot of redundant calls, since only the first instance would actually achieve this
purpose.
▪ Redis would then notify every instance that the “expires” key was deleted.
▪ Every instance would then connect to Redis to get the session details about the
expiring session to handle the expiration.
▪ A little bit of math:
▪ 100 servers * 2500 expiring sessions = 500,000 access at the same second
▪ The more you scale, the bigger the problem

Solutions:
▪ If you don’t care about session expiration events, set the cron job to an impossible value such as
February 31st: 
 
spring.session.cleanup.cron.expression=0 0 5 31 2 ?
▪ You can also disable registering to those expiration and deletion events. Note that all servers need to
disable the registering for this to work: 
 
@EnableRedisHttpSession
public class RedisOperationsSessionRepositoryConfigNoOp {
@Bean
public static ConfigureRedisAction configureRedisAction() {
return ConfigureRedisAction.NO_OP;
}
}

▪ If you do care about session expiration events, set only a specific instance(s)
to handle those events.
▪ On all other servers, disable receiving the events:  

@EnableRedisHttpSession
public class RedisOperationsSessionRepositoryConfigNoListener {
@Bean
public RedisMessageListenerContainer redisMessageListenerContainer(
RedisConnectionFactory connectionFactory,
RedisOperationsSessionRepository messageListener) {
return new RedisMessageListenerContainer();

}
}

To Recap
▪Be prepared to scale

▪Monitor well, monitor with caution
▪Open source rocks 🤟. Get involved! Tweak the
framework for your own use case.

Thank you
http://odedia.org

Spring Session Redis Why How and Production Pitfalls

Caricato da

Informazioni sul documento

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Spring Session Redis Why How and Production Pitfalls

Caricato da

Copyright:

Formati disponibili

Spring Session Redis

Why, How and Production Considerations

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Tomcat 🍪 123-456 = David

Information Security Level 2 – Sensitive

🍪 789-111 = Larry 🍪 123-456 = David

Tomcat Tomcat Tomcat

Information Security Level 2 – Sensitive

🍪 789-111 = Larry 🍪 123-456 = ?

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Any data that needs to persist must be

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

▪At the very minimum, use a master/slave setup

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

▪ INFO provides valuable information (such as instantaneous_ops_per_sec) and can be

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

@Scheduled(cron = "${spring.session.cleanup.cron.expression:0 * * * * *}")

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

Information Security Level 2 – Sensitive

▪ On all other servers, disable receiving the events:

return new RedisMessageListenerContainer();

Information Security Level 2 – Sensitive

▪Be prepared to scale

Information Security Level 2 – Sensitive

Potrebbero piacerti anche

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive

▪ On all other servers, disable receiving the events:  

 Information Security Level 2 – Sensitive

 Information Security Level 2 – Sensitive